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This «anual proTides a aescriptioA> of the deTelppMent 
ana a guide to the nse'of the-assess«ent resources de.Teloped in 
connection with the Par Sest DeTelop»ent, Disseaination, and 
^^Yaluation {DD6E) Punctional CoMpetence Training Erograa. The. 
docuaent concentrates on a user-oriented description of the content, 
Talidaticflx asd use of the final Tersion of the assessaent sjstea- ^ 
DD6E is a coapetence-based professional derelopaent prograa proTiding 
a fiexihle axraj of training resources concentrating on entrj IfTcl 
skills and 3a»o»ledge leguired in DDSE. Chapter 1 describes the 
sTstea,'its design concepts, content, assessaent aetboSs, and oyerall 
use. Chapter 2 prorides a detailed description of prograa planning 
assessaent, inciuding establishaect of the student record file and 
the use of the DDCZ Diagnostic Instruaent. The, derelopaent and 
Talidation data on this instruaent is also suaaarized. Chapter 3 
describes psrpgraa p^rogiess assessaent aethods, discusses general 
guidelines for developing product ratings, and illustrates a student 
progress record aatrix. Chapter 4 is concerned aith existing 
coapetence certification. It describes and suaaarizes the field 
Talidation of a aodel Coapetence issessaent Battery that aaj be used 
to certify skills In "instructional piodcct derelopaent*. Three 
assessaent aethods^re eaplojed; ratings, knowledge t^sts, and job 
saaple tests. Pield'test data indicate that the tests are Talxd m 
teras of content Talidity «nd in teras of ability to discriaxnate 
between subjects with and without actual product 
deT.eicipaent/€Talua-f-ipn experience^ (EC) . . ^ . • . - 
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Produced fay the EAR KEST LABOMDDJOr BR WXmQifL SESEASCH A^JD lEyELOUBJT, 
a piijiic, nonprofit oi^nization sup5>orte4, in part by fends feon the Depart- 
jaent oz Ksal^, Education, and If^lfeie, Kational Institute for EdiKation. 
Ihe cpinioss e;qpressed in this doafaent do not necessarily reflect the 
position or policy of the National Institute Sot Education;' and k) official 
endorses£3it by the Institute should be inferred. » 

Educaticnai &velC5WHit is a nei» discipline. It involves, Sxst, fccusang en 
an iapcrtant a-^d STJCcific area in need of iajJioveasnt, aaa second, iavsating, , 
field testing, .and" validating a. isefid solution to that prcblem or need- Ihe 
solution aay be .a sel£-ccntained product or process to be used by educators, 
by students, by parents, or biy^all of then together. 

In 1970 the Far Jfest Isboratoiy fbaafied a ccnsortiuE of RQ) zg^aaxs and 
educational institutions. Oir purpose «as to develop and field test anrtiuc- » 
ticoal jBodsls laterials and to train cntiy professional and par^J^siiBai 
pcjsceael in cducatioeul dsvelcgjoent- dissemination, and cva^tion 

, Screes. The original «rf>^of the FAR KEH-OKSOSIIlMRaBD^^TO 
in3i3ed Arerican Institutes for Research, Palo Alto;. Educatursal Testing 
Service, Ber3:eley; Far Kest laborator)' fer Educational research and 
Dewlopraent, Ssn Francisco; Hasan Resources P.esearch Orsanization, 3-Ionterey; 
Lodtheed MissHe and Space Agency, Educatiraial Systens Divisian, Sss^Tssle; 
and Stanford Research Lnstitute, Palo Alto, as R5D ag^es; and JSontciey * 
Pttiinsula CoUege,, Hcsntercy; California State University, San Francjsco; and 
Saniiateo College District, San ilateo, as educational institutions. Betxeen - 
1970 and 1973, the Jar Kest Cmisortiua for DD55 Training i«s fended by the 
Research Training Branch of the JCational Center for ?.esearth and Dei-elopDent, 

- U.S. Office of Education. i>jrin2 this period, -ccBpetcnce-based pro^is oz 
DD5E training xere initiated at both giaintt and undergraduate levels^ 

In 1973 the "National Institute for Education assuoad spcosorship of the 
project, lAich was redirected frea dsvelcp^cnt and training to a oeve^cj^t 
effort exclusively. ccsicemed with curricuiua and assessaent materials- Uie 
Far Kesc Laboratory assumed pricar/ responsibility, but ccntinu^ '^t^'^,^^ 
ixntract certain instructionaL godule devclqpBent with the Institutes 
for Research and the Hman Resources Research Organization. Ihe fuial 
^■ersions of these jaodules have Jbeen the responsibility of tte Uboratory, but 
credit for ^eir dewlopnent is right&Uy shared by all aesbers of the Far *- 
"Jfest Consortium for DD§E Training;. Each DD5E instructional aodtlle tas 
coBpleted se\-cral research and de«lopnient cycies prior to release for 
iCTToduction and distribution. At leasts three phases of field .esting-tryout 
of the prototype, a supervised field itest, and an cperational test mder 1 
normal, user conditions without Laboratory participaticai— preceded for^l j' 
external review and an official decision on accept2bility-a-->^^ 
• - * * — 

The Laboratory's mission is to carry out surveys, research, deve^^t, and 
dcraonstiaticais in education,. to dissesdnate infonsation derived froa such 
activities, and to provide services to schools and other educational agencies. 



ERIC 



xn 

is 



During the pasi few years, txainii^g 4^^*^^ 5iave^e«« interested 
-conpelJnce-l^sea- or -pe:^r«=nce-basea- educition.^ (toxeatly, there 
a^br affeit in' teacher feducation and significant- actxvity m ctter 
rrol^i^^ Station areas. Be^ifining in 1970, the 1.est <^-^?r^ . 
ITr DDsfe-Training, a groiip^f several ed^icnal ^-^^"^^^^^^^ 
S higher^^aSon lo-^ted ^ the San P^^oo ^/.^f^' J^^f^^^'' 
-and ■ ieratio«,of a cc^tence-hased pr^^fer ^-^^J^^f 
a«it, SsseMination and evaicatioa CdDSE) personnel. Initial .y, #ie £3OTSorcii« 
foqnied on the develoj-ent and i^rple-entation. of ^ff^^f^L^^^^^ 
leS^rograas that provided for joint planning,- and cnrricul« f^"^^^ 
f^S^^^ of Alleges and ^diversities and by f^^^ ^ f 
RfiD aSncies. When sponsorship of this project was transferred 
Of fi«of JEducation to the Kational Ihstitnte of Education, the F«>3f ^ 
eSSil^fted frc« .cnrrictatm develof-nt and operation, to '-ter^ 
SeS(»ent and- progra. dif fusiorf. By the end of 1975, the prosect cc^l^^ff . 
fS^t S ISTS t«nty-three instructional -odules covering six content 
ilS:^^^, infor«tion%ata collection -^^^V^T^^^^ 
S^int, e^uation,dissemnation and «rXetin|. "^^^f^^^^J^^f^ 
are a^ld b^r several sui^rting.'doct-tets vhich provide^ 
plannSgTijistallation and operation of a variety of configurations of pre. 
service and continuing education ^rogra»s. 

^EhiS docmient descrihes-the'raCB- Assessment Syste-^^e aevelopD^t_ 
and validation of thr^e^types of instruments (ratii^», J^' 
fcrmance simulation tests) vhi<d», represent ijij^rfeanl: co-ponents of the 
assessiaent system* ^ . - * 



The DDSS Assessiaent System described in this report evolved over several 
years and is the product of' a nianber of persons. 

From initiei conceptualization of the project to its coiapletion 
D^. Susan Jaein and Dr. John Sgerxoeier of the National Institute of Educati<^ 
played significant roles in securing support and providing assistance, 
fiheir encouragenent^ patience and constructive criticisa sustained our 
efforts throughout the duration o£ the project. , Taxing the NIE appointed 
site visitors. Dr. J^ohn HopXins, forirerly of Indiana University and now at 
iiesearch for Better Schools and Dr. Eva BaXer, now Director of the Center for 
Study of EvaiuatiOT at DCXa^ deserve special aientioa for the extraordinary 
quality of their evaluations ^nd suggestior*s. ^ 'I 

On the ^ West4^ratory staffs inany persons contributed. Dr. Joseph Ward 
Dr. Wayne Rcfsenoff and Dr. John Hourigan developed and tested the "iiini- 
batteries" whicii served'as prototypes of the Competence Assessoent Battery. 
Dr. Barbara Havassy and Ms. Andrea lash developed and field tested the DD&E 
Diagnostic Instruzaent and also developed the preliaoinary field test version 
cf the Conoetence Assessment Battery. Dr. Laird Bladcwell supervised the 
field testing and data analysis, revised the Battery, developed the scorrag 
nanual and carried out all^^iases of the validation field testing, scoring, 
and analysis. He also contribTf£ed significantly to the preparation of thxs 
final rerort. Ks. Carol Burkh^, the project adsainistrative assistant, 
naintain^ ail project files, 'stiipervised the production of reports and anstru- 
SDents, and dealt .with all of the technical aspects of jnail-out and follow-up 
of the field tests. Ks. Lenni Ball supervised the production of this report. 

We gratefully acknowledge the assistance of instructors and students of 
the following institutions who aided us in field testing one or laore versions 
of the instrusient in the DD&E Assessiaent Systesa: 



Brigham Young University 
California State University, long Beach 
California State University, San Francisco 
California ^tate University, San Jose 
Georgia State University 
Michigan State University 
Tecple University 
University of California, Berkeley 
University of California, Los Angeles 
University of Indiana 
** - 

Azaerican Institutes for Research 

Appalachia Educational Laboratory 

Courseware, Inc. 

Defense Language Institute 

Far West Laboratory for Educational EfiD 

Human Eesources Research Organization 

Northwest Regional Educational Laboratory 

Research for Better Schools 

Teaching P^search 



in addition to tfee comets and critiqoeS provi^ by field.test subjects 
sxt the above iistitnSoas, ve especiallx wish to ac3cno«ledge the assistance . 
of the following persons vh5 have reviewed aad critiqued pxar^, onstraaents, 
or reports at various stages of this projects ^ - - ^ . - . 



Eva Baker, University of Calif omia,^Ios ^Sageles 
3ela Eanathyi Far West I-aboratory -- 
aiaiard Sajfbe, Georgia State University 
Walter Borg, Utah- State University - ^ 

Leslie Briggs, Florida State University ^ 
Robert Gagne, Florida State University 
John Heiif>hill, "Far Wtest Laboratory 

Richard McCann^ University of (^fomia, los togeles 
Hoimrd RcFann, Huten Resources Itesfearch Organization 
Paul Kadarasz, Defense Language Institute . 
David Merrill, Courseware, Inc. . 

Bodney Reed, University of CalifoEDia, Berteley ; 
Jack San^rs, Appalachia EdncatixSnal-I^ratory 
i5el Schalock, feacMng Research 
Glenn Sne2becker, Tenple university^ 

•TO all who gave so much of their tiioe and thought, ay deepest 
appreciation. 
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EXECUTIVE SIMOKr 



. - . Ibis Mnual ^vides * -aeecri^n of the ieTelo|»z«- and * guide to the.- - 
a«e of lie 'assesjaent resources h«ve been develx^ la connection. i*itb 

tlie Pir Ifest Derelbiient, Disisepination and »^»tii>n- lOOSE) Rmctiooal XJoape- 
i:fflce Traipijig Pzogra«.*' . , , -« 

Previous reports have descried the total DM^ program a^ign (Hood, «fe 
19701, the"as8ec;«etot syste. develop«ent plan (Hooi'and Banatly, 1973), ind"' 
inter!, stages of 'progress ^ood^ HavassjT* 1*^ "73,' Mood and laah,- 

1974; H^, iash and BlacJwell, 1974). Jlhis dcx»nt oooofntrate. jariaarily 
on a tiser-oriented description of the content,, validatixii, and m* ot the 
final version of the assessment syste*. . . ^ . 



■ The' Far West vass, Functional, Ctoi5»etenc^ iftfaining *rogra« is a 
.-Tiased profeglohal-develop-ent progrd. 1*^1 provider . flexible ariay bt 
training resources lAich concentrate 09 -ent^-le«l- skills and Jcnovledge 
r^red in educational de^oi^t, dissemination «»d evaluation (COSE). 
Oncpait of the= program's Jffort focused on iihe develop-ent fad valittation 
of an assess-ent system w^clTis-diil^eff t6:ard l**!^^^^ and^instructof- - 

' inr ' (1) pre planning, (2) progress Mjnitorlng, ind X3)-aiit: .is«i«nt,^ 



ta«pter I describes Ihe Ito West DDffi 'ass«s«ii»t i6r«t*, iti^desigp 
concepts, content, assess-ent .ethods, and oVerall use. One of theiM3or 
design cohcepts Is that lie lssess«mt syst- be deoision-^lant^i, and that 

'it be desired to facilitate the variety of decisions, ifhidi -ust be made by 
students. Instructors and cs{>loyers. The assessi^it system ^iwida. a fra-a- 
tK>ik and the assessi^t re^mrces for organising « 3»ighly flwdbl., ix^ye^i-^ 
and practical -ethDd for ^llecting and integrating lnfor«tl«» OmxLvmi ±tom ' 
a ^ety of «>urce.. fte three basic ais«.«-nt -ethod. aza rating^, 

- teats and job sasple tests. 



* »*cmcally excluded is' any trealmnt of tha '^^'^ 
are^rectly Is^xriated with ead» of the 23 instructional pod,a«i. 
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btapter II provides a detailed descrip.tion of program plannin'g assessment-, 
including the estaljlishjnent of the student record f ilfe and the use of the DDfiE 
Diagnostic Instruasnt-^ -This chai>ter junaariz^s the developKjnt and the yali-' 
. daticn data pn thiiinstruzient, whidi*^ also dtescrlbed in detail in flppeiid?2 » 
^e Diagnostic Insffcaent recora of a student is examined in detail to illustrate 
its interpretation and \ise in student program planning. ^ ^ ^ 
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• * - Chapter III describes program -progress assessment methods, bifiefly 
discusses general guidefenes for developing product ratings, and also pro- 
vides an illustration of a sttident progress record matrix, ^ / . . ; ' 

Chapter lY is concerned with, exit coapetence certification. It describes 
and suijoarizes the field validaaon'of a model Conpetence Assessment Battery - 
that may be^Jised to certify skills in the' aiea of -instructional product . 
development.*- Three assessment laetliods are enployed: * -ratings, knowledge 
• tests and job saiif)le testsf^ Field test data, indicate that each of the instru- 
ment^ xk valid, both in terms of content validity and in terms of ability to 
'discrimiiiate bet&een subjects with ariS without actual product development/ 
-evaluation e35>erience. t* , ' - ■ 

. ' ^1 V' ^ 

Even where there are significant correlations between the three type\ 
of instruments, th^ relationships are very modest; hence each instrument tends 
to provide a different 'fcind of valid information. -Since many professional 
training programs seem to rely heavily 'on paper and pencil tests, of .knowledge, - 
the results reported raise questions about the utility, of ^sessment systena 
' -that* rely exclo^ively or almost exclusively on such tests for measuire- ^ 
'nent of knoifledge. However, because the several' different types of assessmenj: 
instruments used in the Far West ^)DtE Assessment System are reasonably indepen- 
dent of another, and yet all are valid, their oo«bined use should increase 
■the overall validity of competence assessment decisions. 

•Rie. %6veral appendices to this document contain detailed information on 
i^he development, validation, and scoring of the individual instruments.. The ^ 
assessment model and the c^ssessment resources deseed in this manual repre- * 
sent 4 K>derately extensive but yet incomplete set of materials. We bilieve 
they are' sufficient in scope and quality to provide a vcsfy good start in 

9 / * ' 
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,establishing*a conf)etence-based DDSe' training cprogram with practical assessment 
capacitor; but there deliberate and clear challenge t^-4ake significant 

additions ot modific^ons to tShe resources provided. For those not directly 
cijncemed With 'assessment of coii5>etencies in educational DDSE, the acdel and 
its validation data may provide* ideas which can he adapt^ to assessment in 
other professioiial axeas. ^ ' 

■* . i 

The manual has be^p written for three, types. of audiences: (1). persons 
with a general int^est in assessment problems in: competence-based eduction, 

(2) persons with specif ic 'interest in the assessment .of RSD personnel* and 

(3) instructors wh<i are adopting or adapting the Far West DDSE Functional 
Competence Traiining Program. a,' - ' - 

For those with general interest in conpet:ence-based" educational assessment 
we recommend Chapter 1, Chapter ll (omitting 'pp. 23-30), and Chapter Ill'r «iis 
will provide a description of the-rationale ^d method. Those with specific 
interest in the assessment of RSD personnel may wish^o read Chapters I-IV, 
plus iipp^ndices A and B,. which describe' the development and validation of 
the assessment ii^truments. Instructors who are -adopting or adapting the 
Far West*3DSEl Functional Coii5>etence Training Program will also need to use 
the materials contained in the other appendices. 
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Clis aanual provides a description anfl guide to the use of assessment 
resources which have heen developed ii. connection with the I>evelor»ent, Disse«- 
nation, and Evaluation (DDSE) Functional Competence Traia^g Progxasu It also 
provides technical information on the development and validation of these a. ^ 
laent resotirc^. 

Oo«5>etence-based education has b^««e.a -movement- t^at offers significant 
;:«chan£sBS for educational refo^. Yet one of the aost perplexir^ prt*le-s 
faced by those developing co=j>etence-hased education prpgra.s is the defimtxon 
and assess=>ent of cc=5>etence. In discussing Perfonnance-Based Teache^cat«,n 
(PBIE)* Elan observes: ^ 

But the overriding prohlen before which the i"^!^^ j^^^^s 

cance is that of the adequacy of ^'^^^^^^^^f^f SfST " 
pSe can only be successful if there are adequate aeans to assess the ^ 

ffjgf not have ^^H^vond current <-'^^^^^^^^ 

■ »n-.._.s new aethods a^ ^ foi^d for assessing the ccmpl ^ 

J ^ ^..^ effective ob1 ^^v^-<^ Whi^ are ^^"^^^^J^ ^f^ 
S r^aSi^. ..Yet this is 'the to unda^on stone on whxch. the progra» 

■ rests--- ^^^^ 21-22, ei5*asis Added) 

David Krathwohl, in an introductory note to JacJc Herwin-s paper («erwin, 
1973), concurs wi€h these words: 

one can predict ttat perfonsance-based tead.er ^f^f^^J^^ ^ 
~^rtSn to fail to reach its ultimate cfcjectrve if it conti5<«s xt» 

«-Perfor^ce-based- and "c^"-*""-' Jilf^l^ 'Srl^Sa^STS^ 

perfora* (Houston and Hovsaar 1972 •) 
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present course. 5Siis failure vili be catised by the aiiaost SysrpletG 
lac* of attention given to' the assessment of teaching cccnpetencies, 
a core concept of P322. Only by snch assessment can ve zdhiBT^ the 
goal of assuring that a teacher can indeed perfona in vays that result 
in children learning. 

laitil srsch assessment can. be siade, aciiieving PBTS will be a xcyth... 

ttese [assessment] problems require great concentration of thought , 
effort and research. Until and unless sorae real progress is mde^on 
resolving the probleiss of instrumentation and saeasurement, V3TS vill 
go dawn in the history booXs as one rore bandwagon in the long line 
of oyer--si3rolistic solutions for complex prcH>l€28s. 

{Kervin, 1973, pp. v, vi) 

Aithou^ Elam and Krathwohl refer specifically to teacher educaticm, their 
observations are equally pertinent for all cosMtence-based or performance-based 
education. Specific details of the assessiaent problem snay change, but not its 
general character. 

•Qie instructional and assessiaent resources developed by the Tax Vest Censor 
tiua for Educational Developtient, Diffusion and Evaluation (M)SE) Training were 
designed to be esployed in preservice training or continuing education of entry- 
level professioniOs- As Elan predicted, nost of the Far West Consortium's 
effort went into the developaent of instructional saterials, developing and 
evaluating new insti-uctional systems, and thf like. However, because we ea- 
plc^sd repeated cycles of developeaent, testing and revision of DDSE instruc- 
tional rpdules, the development of aodule tests was an early concern- Serious 
attempts to develop a conpifehensive assessnent systen came at a-Tauch later 
stage in the project. We found the task to be inaensely aore difficult and 
costly than we first suspected. When confronted with the choice of preparing 
a coBprehensive set of assessnent instruments which we couldn*t afford to 
validate or of rigorously developing and validatUig: a auch smaller set of in- 
struaents, we opted for validation of a "model" consisting of a core set of 
assessnent items. We have atteapted to provide guidance for the instructor 
-who «ay (and should) want to prepare additional itees following the aodel- Hbs 
result is an "open* system. ^Hie assessment nodel and the assessaent resources 
^provided represent a «3derately extensive buf incc3i>lete'set of aaterials. Wte 
'believe they are sufficient in scope and guality to provide a very good start 
in establishing a coi5>etence -based DDfiE training program with a practical assess 
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Bent capacity; but there is a clear clialienge to naSce significant additicos or 

Eodificatixsns to the resources ve have been able to pr&vide. • 

i 

B. S0CP5 fiSD FOCCS CF ISS FAH CPgB C3C«?SgESCa ASSKbisy^g SYSSSjj 

1. Definition of Conyetence 

Oirrentiy', there see=s to be no cornonly-accepted at^roach to cevSlooiog 
co=5«tence assessnent instrunents; in fact there is not even any ccEsaonly- 
accepted definition of -coa5>etence." Schalock and "Sioras (1973) have laaoe a 
useful distinction between two laeanings oosconly e25>loyed. Qae equates coas>e- 
tence vith the nastery of ^owledges and skills assumed necessary to perform a 
particiilar function- -The second holds corroetence to be the deaonstrated ability 
to bring about outcoaes specified in a given job description. 

2, Criterion levels 

idchard Turner (1971) has provided a finer discrimnation vith six criterion 
leVels. Ihe siy levels range froa demonstration of oastery of knowledge and 
understanding (in level 6), deronstration of skill attaimaent in sis5>le training 
and laboratory conditions (in levels 5 and 4), behaviors- in actual conditions 
(in level 3). to evidence of short- and long-tern pm)il cihange (in levels 2 and 1). 
AS we ascend Turner's criterion ladder, the prdblens and costs of assessment rount 
rapidly. 

in the DD&E assessmenc systen, we provide instruKnts at Turner's three 
lowest levels, naiaely at level 6, tests of knowledge, and at levels 5 ^and 4, 
performance of skills in job sarnie tests and sianilations. Assessment at level 3, 
behavior in actual conditions, is provided only through ratings by si5>ervisors. 
Thanks to recent work by Pophaa and others (1974), we aay so^y have instru- 
mentation appropriate for measurement of short- and long-tera effects of a 
developer's products on target audiences in levels 1 and 2. However, since 
test deveXopient and data gathering costs are so high at revels 1 and 2, we 
liave not attenqpted assessment at these erf ierion levels^ 

It nay be helpful to look at the cos5>etence content area addressed by the 
Far west project. (See Figure 1, adapted fr«n Clark and Hopkins, 1969, p-14.) 
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Fi«3"re 1 locates the scope, focus, and area of concentration. Oar foras is <si 
corj^tencies required of entry-professionals {inasters degree level personnel} 
in the area of developoent, with soiae spill-over into irnediately adjacent areas 
of r^earch and diffusion, ^e reason for this focos is that it corresponds to 
the area addressed by the training naterials ve have iSevelqped. Figure 1 indi- 
cates that the scops - the area vhere the assessoent =aterials say be useful - 
extends to a wider range of professionalization levels- Si25»iy stated, althoac^ 
the co=petence criterion levels focus on coi^etence ar5>ropriate for entry-level 
professionals, the assessri^at instruments nay provide useful infersation about 
levels of corpetence which are below or slightly above those considered appro- 
priate for ros entry-level professionals. Finally, Figure 1 indicates that 
we oade a concerted effort at intensive instruaent development and validation 
in the sore restricted sub-areas of "engineering padcages and ^ro^aas for 
educational use" and "testing dnd. evaluating solutions and progifans." 

Figure 2 indicates that the project, in its attest to derive curriculua 
and-dnstructional objectives, has dra>m on three data bases, the AEHA Task Force 
on Training (Wgrthen, et al., 1971, 1973), the Oregon Studies in Educational 
EDDSE (Scialock et at., 1973) and the Far »est Consortiua's task analyses (Hood, 
et al., 1970). "parallel developjaent and field testing of instructional aodules 
and of test items provided field test data on students as well as expert review 
of content. This experience was fed back in revision cycles resulting in aodi- 
fication of both the instructional resources and the assessment instrumentation. 

C. DBSIQt CCgtCSPTS 

one of the laajor design concepts in our approach is that the assessment 
system be dacision-ori^ted. He are not so jnuch concerned about tests and 
their psychcnetric properties as we are about the utility and feasibility of 
co«petence-based decision making. Who are the decision .akers in a coBf>etence-- 
based edticationaJ. systen? - ' ' 

First, there are the students, " ihey are concerned with questions, regard- 
ing' status and progress. Ihey want to know-what co:^tencies they have already 
mstered; what their level of proficiency is ^ds-a^E draining exit or cuf^loy- 
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Figure 2 



Derivation and Validation Linkages 
for a Catalogue of DQiE Coapetencies 
2 Coopetence Assessaent InstncDents- 




iDtiii requireiaents; and vhether they should repeat a uait of ir^ctriiction or < 
proceed to the next unit. 

second, there are the instructors. They want to fcoow the students' carrent 
levels of ac=055»lishaent and skills; what further training should be planned; 
vhat progress is being sade; and i^ether in fact exit aastery is attained- 

•IT 

A third group are the esployers. Hhey are particularly interested In the 
natch between a prospective enoloyee's attainisents and the job reguireaents. 
Does the person qualify for a particular job or will further on-the-job train- 
ing be needed? Given valid, and detailed infonaaticn, ei^loyors «y restructure 
the work for sore effective "use of their eaployees* Current levels of competence 
or possibly nay encourage aore on-the-job training or inserwice education. 

Finally, there are those of us who are concerned with the developoent 
and evaluation of coapetence-based trailing: prograas and resources. 3hey need 
to know about the range of entry-levef knowledges, skills, and sensitivities. ^ 
DO specific instructional materials and aethods 'facilitate attainnent of partir- 
cular coapetencies? How can instructional materials and methods be improved? 
HOW can credible evidence of program perforBance, sufficiently i^essive *o 
persuade potential users to adopt the program ox to hire its graduates, be 
provi'ded? 

oSiese are soae of the questions that students, instructors, eaployers 
and developers say ask. How are the answers to be provided.? 

We ha,ve already indicated that one of our najor design concepts is that 
the assessment system be4§cision oriented. ^ Obis in turn iaplies that we be 
concerned with utility, usabil ity, validity, and reliability; and in that order. 
If the instnaentation does not lead %o better decision making, for all the 
decision «akers, there is little chance that the ^sessi^nt syste. »411 actually 
be implemented and even" less chance that it will ie maintained. So, above all, 
*the system must be useful. It must iave apparent and real utility in helping 
users.inake better decisions, ^e usability of a system means tiiat time, cost., 
psychol^ical threat, etc., must be minimized while maximiying the information 
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obtain©^* for each stakeholder — stilSeat, instructor, potential or actual esployer. 
In our opinion, the reqnireiaents for utility and usabiUty transcend those for 
validity and reliability. , Obviously, there past be scae non-trivial level of 
validity and reliabi-lity- But if one takes a decision-theoretic approach, it 
is the utility of the decision that is paraaount. l5ie-najor- issues are the 
risks of error and the costs of those errors. 5he challenge for us has been how 
to configure a set of infonaaticn gathering instrfenents and processes in a fora 
that would lead to their ^ccfeptance and use and also provide for efficient organ- 
ization' of information that would have an actual bearing on decisions. 

Utixee issjortant and highly practical, technical pointe are thats 1) we 
are dealing with a classification sodel; 2) we are dealing with a saltivLB^ 
sequential defcision msdel; and 3) a rniltiple-sequential decision nodel aay be 
best handled -with a Bayesian statistical decision approach. 

Many of our assesssent approaches are based on selection nodels where we 
attearot on the basis of a one-€ine assessment to decide whether a student 
should be selected into or out of a training prograa, passed in a course or 
certified for graduation. Although selection reniains an aspect of coBoetence- 
based programs, the assessment probleas really de^ ©ore with classification, 
•ae instructor, the student and the employer have different perspectives and 
the majority of their decisions are not simple -go, no-go" decisions. More ^ 
often, it is a matter of deciding how well prepared, which job, what kind of . - 
a career, how much and what kind of supervision, whkt potential for a^dvance- 
ment, what kind, of instructional resources, etc. 

The" coiietence-based, individualized approach provides the need and the 
opportunity to make a sequence of tests and decisions. Few of thea are totally 
irreversibly- This is a fortunate situation 'since few of the measures which are 
feasible pri^ide highly reliable or hi g hlv valid measures v^en used singly or on 
one occasion. He Jcnow from decades of personnel research work that interrater, 
reliabilities of coE5>lex performances and ^oducts aay not go «uch above .6 or 
- .7 in actual^ractice. . And validity correlations of a" specific predictor to 
a sE^cific criterion are often in the .2 to .4 rtoge. While this situation is 
tolerable when dealing with groups of persons, it becomes less so when dealing 
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vith decisions specific to one person. However, if ve ta3:e a relatively large 
n^r of measures over ti^ and e=5»loy a variety of «tbocs, the omolative 
sequential decision process itself can attain a =uch higher reliability and 
validity with correspondingly lovar classification errors ai^d costs. 

we have knci.-n for so=« tix=e that, at least in theory, the Bayesian statis- 
tical approach was an attractive alternative tc the classical aj^roac*, since 
it is rarely the case that any decision ^er has a flat prior e^ctation vxth 
total ignorance regarding the probabiXitles of alternative outcoiies. Recent 
contributions by Brown (1969), Ferguson and Bovick 11973) S«-inathan, Hanbleton, 
Algina (1975) , and others, suggest that eventually we shall be able to provxde . 
re-.atively ^i=ole procedures for all decision »aJcers, whether they are stulents, 
'i„s;ructofs, or employers, to arrive at =oro effective decisions through fuUer 
use of available infbnnation. 

POr the above reasons, we cho^e to focus on the idea of developing a bighly 
flexible, inexpensive and practical as^essn^t sj^ste^ rather than a sample col- 
lectxon of test instruznents. Indeed, this perspective caused us to be far 
«,re concerned about herping decision na3cers to use info«»ation «,re effectively 
than in singly developing nore reliable and. valid i»easureBent devices. Please 
understand that 'we are' not underrating the need for better .^ure«ent, but 
si^lyKasserting that' froB a very practical point of view, the crucral proble» 
in Wtence-based education is to get people, including instructors and 

^ .....on^s. to w^t and know how to use assessment infqrmtion, UntxX thxs 
■happens/w; have technically i^c<^le but ^usable, competence assess^nt 

' instrun^ntation. Hence, the objective was to .a3ce .the pr^ced^n:e^ - 
ines^nsive-, eas^to use, easy~^ understand, face valid,>e~fotr'and ^aningful. 

1. Three-' Maior Hetl?o^\ - i * ' ' . , 

After review of a r.^r of alternatives, J selected three basic assess- 
-«ent methods.' 'ratings, Knowledge tests,, and jpb.Wc --sts.* Although these 

initially we also-in tended to deve^p g/e^^ ge^t^Tf S^ral 
-products". Hcwever, e35>erience based on f ^ 'duce Wous 

virsion^ of simulation tasks, all '^J^^^fr/ J^L^gl^^S ?e^turc, 
"products", has thoroughly discouraged us in this pa^cux^ 



ERIC 



22 

9„ / 9 



/ 



nethods can be augiaented by biographical infoniatiCT, questionnaires, inter- 
views, and other sources of inforration, these three methods constitute the 
foundation for assessnent. . 



Ratings 



we have found that «hile ratings by students, instructors, peers, and- 
work s»5>ervisors are useful; possibly only the student rating may be feasible 
in the preservice prograa- Although subject to vell-Jmown types of error, 
extensive e=5)erience in the use of ratings indicates that if well-designed 
and properly employed they can yield rodestly reliable' and valid information 
at very low cost. After trying several forsats we selected a descriptive, 
performance-based rating scale vhich perxits relatively unambiguous detenaina- 
' tion of observable or easily infei^le levels of perfofaaance'- 

3. Jca> Knowledge Tests 

Initially, we attempted to develop highly objective and easy-to-score 
' job knowledge tests. However, we found it difficult to write ites? testing 
for conprehension, abUity to i=ake applications, or evaluative judgiaents with 
- conrpletely objective formats- Our selected approach use essay and short- 

" answer written questions, which call for rore complex constructed responses on 
the part of the testee and also for greater thou^it and attention on the part 
of the grader. ' -this is one area where we have accepted increased gradi-ng «Jsts 
in the hope of obtaining greater meaningfid-ness and utility. 

4. Job SansplB Tests 

Ratingi and job knowledge tests are relatively easy to develop and can - 
c^ver a lot of "coiopetence territory- in relatively short tiiae..^ Job samples 
are soiaething else-' Generally, they are quite time consuming and e35>ensive. 
, to admimster and to score.- Moreover, it is quite difficult to avoid a certain 
degree of situational specificity which may significa^tl^- limit the generality 
of results. After considerable exEferimentiTig, -we developed a simulation test 
which consists of a series of separately scorable but logically related job 
saisples « 
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E. OVEKVIEH OF ISE OSS OF T R^ aSSESSHEHT. SYSIZH _ 
- t ■ 

'our concerns for costs and practical use, led to a ^ential strategy in 
whic* the least expensive devices such as self-ratings are used »ost extensively, 
but with cross checking against i»re e^ive sources of infbr«tion such as 
knowledge tests and job sa=5.1es. ^ item saaoiing strategy co,5>led with a . 
Eayesian decision approach in which test iten^s are selected for their relative 
. potential in reducing the decision oakefs unceriainty is e«i>loyed. 

The various types of instruiaents and their use are suggested in Figure- 3, 
Please note that we have created an artificial di-<3w.to«y tq illustrate a 
difference between diagin^stic use and assessment use. In the earlier stages 
of the DD^ Training Program, "the «^hasis is pri^rily on progra. planning, 
guidance, and counseling. As the student progresses, the e^hasis tends ta shxf t 
to concerns about co:r5.1etion of «>dules, attain»eht of objectives, credit for 
at^^inH^u, ar.d -Lti=ately to graduation aid co=s>etence certification. 

''oat:5.arable data, and soineti^es identical or parallel test items and in- 
strunents, may be ei:5.1oyed for both uses. A kudenfs file is -opened" by 
recording pertinent biograi*ical, academic, ind work e^rience data from the 
student., application form. This may be aj^nted by instructor interview data, 
me first structured instrument th^ ^d^: :&ounl:ers. is, the Diagnostic Instru^ 
■W of 72 self-rating it^ms.which has been designed-to reveal a coirj^tence 
profile on the three DDSE functional contexts (develof^t, 'disse^tiori,- and 
Evaluation) Lnd on six process skills (analyzing, planning, producing, evalu- 
ating, collecting and orgahizing information, a.nd co-nmicating) . _Pn the basis 
of this information taken toother (remember we are e^loying a seguentxal, 
Bayesian, item sailing strategy), the instructor decide to pr^ are^ 
where further informatioA is needed - perhaps, because the stud«t»r self 
appraisals in those areas suggest 'discrepancies in either being 1>igher or lower 
than expected. The least e^ive alternatives available are: 1) inte^ndew,. . 
2) ^e of a second tailored self- rating drawn from the DO*E.it« pool or the^ 
t^e of one or .ore of the DDSE -odtile tests. In some inst^s, .«t*ervi»or.^ 
ratings, jc* san^las or prod.^s -ay be' available. However, their ^'ir.: , 
diagnostic situations 'would be ungual except in continuing education progr»e^ 
• or in.progra«s prSkdTng simultaneous intern or on-the..job trying. 
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FIGURE 3 



M>&E ASSESSMENT ll^TFmStTr USE 



INSTRUMENT 
TYPE" 


DIAGNOSTIC USE 

• Guidance . 
« Counseling 

• Program Planning 


ASSESSMENT USE 

• Prooress Assessment 

• Credit-l^-examination 

• Certification of 

Mastery 


1* Biographic Data 


DDfiE BIOGRAPHIC 
DATA FORM 


(Same biographic data) 


2. Interview Data 


Instructor's initial 
program planning 
interview 


Instructor's interview 
file 


3." Academic Data 


Student's application 
form; admission 
record 


Program progress 
information 


4* Work Data 


DDSE BIOGRAPHIC 
DATA FORM 


S'..udent intern or wcfec 
application data if 
applicable 


5. Ratings 

a. Self ratings 

b. Supervisor 
- ratings 

c. Instructor 

ratings - r 


DDSE DIAGNOSTIC 
INSTRUMENT 
(DDfiE ITEM POOL) 
' (Use supervisor sceile 
with Diagnostic Instru- 
ment if supervisor rat- 
xng' can be oDtamea; 


DD&E ITEM POOL ' 
DDSE 'ITEM POOL 
(supervisor ratings if 
applicable) 

DDSE ITEM POOL 

*" * 


6. Knowledge Tests 

r 


1 

DDSE MODULE TESTS 
may used as pre- 
tests if indicated 


DDSE MODULE TESTS * 

DEV£LOPM£Ni- oAixatix 

KNOWLEDGE TEST 

Instructors Tests ^ 


7. Product Ratings 


Instructor appraisal 
of DDSE products or 
reports student may 
have produced 


DEVELOPMENT SIMULATION 
TASKS 

Instructor's ratings of 
'student * s products^ — 
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AS Students progress thrduyl. the prograa they will encounter vp' to twenty-' 
three ena-of-4odule knowledge tests. Bie students oay also prodacfe a variety 
of rateable products as part of application projects. In soae instances, they 
my have an internship where a valid jc* supervisor rating can be -obtained. - . 

- / At the end of the program this cuBulativer file of 'infonKtion dan be 
augmented by a final self-rating. Because of the flexible 'i'te. saipling fonat, " 
this final self-rating Bay be tailored to probe 'areas coriresponding to the 
student's own program cbjecLives. Vhen relevant, sti>ervisor or instructor 
ratings nay also be obtained. Finally, for those students %Ao ha« focused on 
development and evaluation, we have provided a acre co-prefaeiisive job knowledge" 
ttest and job sanple (siwilation) test. * . . _ 

Validation was approached froa two directions. 'First a panel ofea^^r-^, 
including DDSE Work supervisors, made judgments about face faiidity and relevance, 
second, we required that the laeasured differences between grqup^ known to have 
markedly different competence levels be significant and of practically meaning- 
ful r^gnitudes. Ihe tedinical appendices to this guide report the results lof . 
our validation studies. 
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II. PBOGKAH PU^miUS 
A. I!gTaOI?gCTICgg 



A fuaaar>entel al^irig^ti^ "6f the DD£E Functicaal Ca^tcnce Training Piogra», 
no matter what its cpeitional configuration (see the I»SE Program Instruction^ 
Guide) sight be, is that the I-rograa vill be characterised by sooe reasonable 
effort to provide ^ individualized program of instruction tailored, to the parti- 
^ needs and interests of e^ch student. Moreover, the Ercgrari deliberately 
^fies the role of the instructor from that of a -presenter - of ih&raatxon 
to that of a -laentor- who counsels and guides students in planning their prograa 
<if study, assists th^a in selecting and using instrucrtjonal resou-.-ces, and aids 
thea in assessifag and evaluating attained coripetence- 

Uttis is a role which calls for use of ^tly more iiiforaation about the 
status 'aiid progress of each student than is the case in inore traditional in- 
structional systems. Given the realities of intoictor ti3»e and effort, there 
are obviously severe llinits in^tenns of what the instructor «n reasonably 
accor5»lish in developing, maintaining- and using su^ informtion. ■; We have trxed 
to keep this in Bind in designing the assessment systea. 

our fixst advice is to create a separate file for each student. . 'Sne file 
nay event^lly contain the following items: 

^ a sunrary status sheet on work accoa5>lished 

, a suamary status shiet on estimated x:a^etence levels 

iiis^ructor's notes (initial intervieW. subsequent interviews, 
coBBents, appraisals, etc.) 
. the progran adinissLns records (application, ^'J^^*^ 
resullS, letters, etc., depending on the re^eaents of the 
'institution an3 departjjent) ' ^ 

copy of the T3iagiostic Instrument record ^ 

IfflCE Tridning Program Biographic 
. end-of-aodule tests, job inowiedge tests, si«ulation tests, and 

otiier assessaent records - 
. copies of reports,'paiers, project descrfptions, etc. docu»entiig . 

student Work ' ' 

letters^ ratings,. and other .infor«ation'8i«5»2ied by intern 
. • ^^^Z or ^Z^s If theprogra«.provid.a. an appUcataon cc^en. 



Because such a file can beccsae quite voluainocs, the two status sheets 
are '.f particular ianportance since they can provide, at a glance, an cverviea 
of where the student is .and what activities have been accossplished. Our 
assiirotion is that the instructor will refer to this file as needed in con- 
nection with eadi progran planning or prograa review session and that the 
instructor will laake at le^t brief notatioi s in the instructor's notes section 
on a frequent basis and certainly in connection with every planning or review 
session. 

?$pendLx _F_ contains i»del copies of several forss. Because we have 
envisioned a flexible prograa in which the Far West DDtS aaaterials would be used 
in coiyunction with other zesomrces, it snist be ajKwasized again that all the 
assessoent forns and instruments whicih are provided should be treated as aodels 
or outlines that should be aodif ied, and in nost cases e:q>anded, to reflect 
the specific characteristics of each institution's instructional nrograa. 

3. THE DDSB COMPSTSiaS MRgRIX 



In .assessing current status or planning for-further study, a basic proble. 
for both the student and the instructor is now to view the dooain of DDSS co«5>e- 
tence. Tne flSRA, Oregon" Studies, and Far West data air indicate that the DDSE 
donain is con^plex and consists of a nmber of different ^eas. .However, there 
is evidence that <jne c^ .discriaifiate between -devel opcenc , - -disse mination,- 
and -evaluation," at least in the' sense that one or sore factors can be relatively 
, unambiguously associated with each of these' ^OTanonly-recognized functional areas. 
•Hie Far West data wouia suggest that, at least in teias of self ratings of levels 
of coa5>etence, evaluation is the laost houogeneous area and develoj^nt is the .ost 
heterogeneous- Stated nore exactly, ODst of the self-rating^ i±tms referring to 
evaluation tasks load, on only one or two factors while ite«s describing develop- 
aent load on several-^different factors. 

The original design for the DDfiE Prograa (Hood, Banathy, Ward, et al-, 1970) 
assxmed that the contexts of DDfiE would be significant. However, we also ass«ed^ 
that competence in all three DDSE contexts would depehff on aore generic skills, 
such as collecting and organizing infornation or cconmicating, that would be 



28 



15 



essensiai fcr effective perforsance in any context- Consequently, ve developed 
a strLctural latrix for the organization of the DDSE rnnctional Context Training 
Fzograa «5ji<A consisted of both contexts and skills. (See Figcre 4.) 

various versions of this iratrix have had different cffifbers of, and labels 
for. rows and colirsns as we searched fox a^qpriate categories to define context 
and skills. 5he IjUSE, instructional resorarces have been organized, aad inter- 
related, to reflect both the context and the skills asproaches, consequently, 
we attei=>ted to develop an assessrent systea that vonld be coigatible with this 
dual orientation. 

Because the ±nitial focus of assessment efforts will be on Program Planning, 
the instroctor must have a good grasp of the overall instructional prograa. 

•ae Instructional Systen and its rationale are described is. other DDSE 
docur^ents, but it =iay be helpful to review a bit of its structure at this point. 
Ihere is a total of 23 instructional iDodules organized in six =ajor series, " three 
series are primarily context oriented and three are prirarily skills oriented, 
although the «>dules have been deliberately designed to afford considerable 
flexibility regarding the nuinber and sequence in whidi students oay use the 
■ nodules in their prograa of study, there is a general plan. 

Ihe Plannihq Series provides an orientation for the ^tire DDS2 Program, 
since it entails consideration of all the phases and contexts of a development 
project. 5he nodules in this series focus on the earlier phases of development 
"(e.g. specification of catcozaes and consideration of alternatives) and also 
' (optionally) provide introductions to the three na-jor^ context series: 

• Developpent ; 

^ Field ,Test and Svalization 

^ Disgemihation and Haurketing 

Itost of the (generic, i.e., <5ross-context) skills (e.g. , analyzing, plan- , 
nix.g', producing) are treated v^«mL the contexts of each of these three series. ^ 
However, in addition to planning skills, two other generic skills areas were 
selected fot intensive treatment: collection and organization of infor^tion 

29 
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Figure 4 



A Stnjctajral Katrix for the Gicanization of the 
DDK pjnctional Context Training ProsJ^ 




Collecting 
Infonnation 



Analyzing 



I 

Planning 



Producing 
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and data, and cvint'-' n ^^-^'on {tjral and iffittea including preparation of reports 
and instructional writing)-* 

It should be noted that althongh the organization of the instructional 
"se^i^ has single out three generic skills areas for special independent 
treatasnt, all the process skills listed in Fignre 4 are included in the 
content of hoth the WSZ instructional and the SDSZ assessaent resources.. 

Khen we ai5>roa(died the task of designing a device to organize infor- 
jaation about sto3ent entry-level cosrpetence status, it was thus natural that 
j#e turned to the aatrix of contexts by skills. 5he aatrix provides a con- 
venient and useful sneans (a) for organizing assessnent inforiaation, (b) for 
relating this infoncation to training objectives in order to deteradne the 
-discrepancy- profile betafeen current and desired levels of cocpetence, and 
(c) for relating these discrepancies to the instructional resources provided 
by the DDSE orograa. 

■ Before exmir^ng the sore complex task of eiroloyiisg several different 
sources of infocaation, let us focus solely on the DCSE Diagnostic Instruoent- 

C. 1SB EOSB DlfeGKOSTIC IKSTRCKSHT 

Few students entering the DDSE Progran can be e^cted -to have any- ' 
thing approaching a clear idea of the scope and organization of the DDfiS 
Coi5>etence dranain. Indeed, there really isn't a very good consensus anong 
DD£E practitioners. But, both the student and the instructor need soae 
reasonable basis for developing an instructional plan. 

Bie M)tE Diagnostic instruaent sakes «hat seems to be a pasonable premise 
that the student is able to sake soiae'rough judgoents about what he or she 
knows or can do with respect to relatively specific tasks, e.g. 

-Write copy for instructionaa laaterials fro« product specifications. 

*Job and task analyses indicated tl,at cos?>etence in these ti«o areas are e^peciaUy 
i«orSnt for ent^-j>rofessionals. Althou^ familiarization ^^J^^^f^^l 
•S^rovided with each context series, it is not feasible to provide sufficient 
attention' to these significant skills by this means alone. 
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-Deteraine if theoretical assmptions cnSerlying various statistical 
tedbniques have been violated io the analysis of data.* 

-Identify the crociai c3iaracteristics of a target group vhich nay in- 
fltjence the dissesdnation effort." 



ae student is asked to rate these itens in teres of the fellcwing 
scales* 



4. 



6. 



7. 



Level of Proficiency 

1 have no specific teowledge about this activity 
xior eiqperience with it- 

I have read about or seen this activity performed,- 
but have no experience with it and don't really , 
understand i.t« 

I have studied this activity or have frequently seen 
it performed and have a good understai^ing of it, but 
I have not yet done it. 

J have a general understanding of this activity and 
have had soae e35>erience with it, enough so that I 
can do it if X have either detailed instrncti<»s or 
close supervision. 

I have enough experience in performing this activity^ 
to do it if given enough <?eneral sts>ervision or 
general instructions. 

I have enough 3cnowledge and eaqperience with this 
activity so that 5 can perfoni this liask quite sat- 
isfactorily without sijpervisicn or job aids. 
I have had extensive eacperience with this activity, 
and can perform it quickly, efficiently and do a top 
quality job. 



ae DDSE Diagnostic Instrument consists of 72 items that are rated on this 
scale, ^e items are organized in blocks with content reflecting the matrix 
structure suggested in Figure 4, with lAree sets of 24 items ^ for the con- 
text areas of (1) developiaeht, (2) field test and evaluation, and (3) ^^dissem- 
ination and marketing. 



*ln various field tes ts, we have used 5-, 6-, and 7-i>ol«t scUea, ^^^^g^* 

of fa^ledoe andlf perfor^mce have displayed extrenely hig^i ^"^^^H'^" 
!l) ^rr5 so«e jt Sification for the confounding that occur, in this scale. 

19 



Ihe instxiUDent record fors, reproduced in reduced size, appears in Figure 5. 
Content of the iteiss is abbreviated on the record forza i3nt does provide the 
stodent and instructor i#ith a coin^enient and specific reference to the content 
of each itea. liie fom provides for the a^n iput at'on of averages for cells ^ rows 
and coluisns. Ihe record fona thus provides a coii5>act and iaoediately inter- 
pretabie suimary of the student's self-assessaent for a system tically constructed 
sarple of DD&E tasks. 

^e record forza can provide a highly useful basis for the student and in- 
structor to discuss r 

• the stx>3ent"s perception of current level of coH$)etence 

• vhi<^ appear to be the areas of greatest- and least coroetence and 
what this izsay inean in terss of the students training,. e3Q>erience, 
and aspirations 

• e^lanations for any unusual patterns (e.g.,. items or sets of 
itens that have been rated particularly low or high relative to 
(1) the student's overall profile of ratings, (2) patterns seen in 
other students with siriilar education or experience, or (3) what 
would be expected given specific inforsiation about the student's 
training or experience 

• what the student perceives as desired xevels of crapetence in each 
context or skill area given the student's instruction prograa 
objectives 

• what the instructor would advise as to desired levels of cosapetence 

• which cosmetence areas deserve special attention in planning the 
student's program of instruction 

How, for some consents. First, please keep in mind that the Diagnostic 

Instrument- Record form is only a point of departure for Establishing a "Baye^ian" 

coapetence record. We have used the label "Bayesfan" in quotations to signify 

that while we are not asking the student or the instructor to employ actual 

Bayesian cos?>utations (See, for example, Novick and Jackson, 1974.) we are — 

asking both to eE5>loy a weak form of Bayesian inference- 

In raking any decision about the student's level of coB5>etence, other than 
-iir the case irtiere it can be xeferred^to-a-highly specific, well-defined,, aceadily 



and reliably measurable, unequivocably "valid," and context invariant perfomance 
{which is highly imlikelv) . -we have to mak e inferences.- Althou^ it is a gross 
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oversiiaplification/ the thrust, of the Bayesian approach is to incorporate •prior" 
fprevxonsly available) and •collateral" (concurrently available) infomation 
with specific "current* infomation in raking an inference le.g,, regarding 
individtsal proficiency level in soae EO&E s)cill or set of skills) rather than' 
basing the inference on the "current" data only. 

One laajor purpose of the DD&E Assessaent Systea is to provide both the 
student and the instructor vith relatively sijcple laeans for collecting, organ- 
izing, and eiploying such prior and collateral infonaation at each significant 
decision point in the student's instiruction progrars. 

Consider for the laaaent the data that saight be relevant to the record in 
Figure 5* Assunie that as an instructor you have worked with*several dozen 
students discussing the aeaning and interpretation of such records, and then 
assuae thac you are 4iscussing the record with a particular student* You would 
have several isiportant kinds of prior infbnaation including: (1) what the self 
ratings of other students looked like, (?) other inforiaation such as courses taken, 
grades, work €«>erience, course performance, etc., on these other students, and 
(3) similar information on courses, grades, jetc. for this student* Tfhls prior 
infonaation can and should be ecrployed in asking whether the student's ratings 
Are consistent with this prior infdriBation. Also, assuming soeae re^tionship ^ 
arong the skills, it is also possible to use the collateral information contained 
in the other items .of averages for other groups of items. a?hus, if the student 
self *rated relatively low on rost items', . and this was consistent with the other 
background information on this student and relative to other students, bW the 
student self rated much higher in one area, one might suspect that this rating 
was due to the unusual interpretation the student made regarding the itens, to 
carelessness in marJiing, or perhaps to relatively higher coE5>etence in a part- 
icular item or set of items. In any case, the situation would call for inves- 
tigation which might include any of the followingr - 



asking the student to discuss _ja LeLj::eajS Pn f^.. for rati Tig tha i4:ea3^^ 
the recorded levels . 
probing the student for relevant information on oast exiferience 
or training that would confirm or disconfirm the ratings 

35. 
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. asking the student to rate additio/al iteiBS Craich ?ay be available 
in the itea pool or vhich could bef prepared by the instructor) 
reflecting related tasks in the s^ general coapetence area 

. asking the student to cosrolete several* knowledge test ite«s free 
the knowledge test itea pool {available for *>st of the areas) 

If the results tended to confina th4 anomalous rating, one could have 
reasonable confidence that the student's/ self-rating was correct. 

Obviously, a siisdlar procedure ^vld be iudicited in the event *that= 
^^ar the overall level of reported proficiency vas une35>ectedly high or low, 
(b) specific conpetence areas'! (sets bf related itess) were unea5>ectedly high 
or low- 

we are not suggesting that €h^e procedures be drawn -out into a protracted 
sequence of sequential testing whidi could easily «d«ust the patience of both . 
the student and the instructor. . But we are trying to «ake at least three 
points: , 

. 5he DDfiE assessment system provides aethods and resources to aake 

follow-tro assessoent when this may be useful. 
. nhe task of interpreting the Diagnostic" Instrui«;t data should be 

approached -with sudi a "Bayesian" perspective about the of^l 

rS^ably relevant data in making an^ initial onferences^about 

student cooipetencei 
. This general method should be employed in the subsequ^t use of 

nei isLsment information that will entfer students' files as 

they progress in^theix Eo's2 training programs. 

D. VALIDATION TEST' OF TOE nTaraiOSTIC INSTRTOSKT 

r - ' • 

1. The Field Test . ' . 

Tables 1, 2 and 3 sumarize validation data collected on a total of 78 , 

subjects representing a wide range of e^rience and training. Table 1 describe, 

- t h. composition of the test population, ^le-2 reports item «ah^ and aign.£i. 



'cance tests for each of the. 72 items, - Table 3- reports scal^an., standard 
deviations, item means (scale means divided by nunfoer.of it^ in the scale; , 
these item means permit direct con^arison among scales ind to'the- proficiency ' 
level rating ^esc^ptio'ns), correlations among scales, and factor loading* ^orx 
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mSZ DaGSOSnC IKSTRDKEOT VflLUKKCa SOBJrXnS 



Muajber Description of Validation Subjects* 

U Students recruited froa a local university whose backgrorads 

were sirnlrKV to that of students we ergcct to enter the DDfiS 
progran. Hiey were seniors or graduate students jsajcrxng an 
education or snajoring in *he behavioral sciences with an 
interest in educaticm. 

51 Gradua te Students (Masters and Ph.D. candidat^) enrolled in 

educational research and development courses. Tnese subjects 
were recruited by their professors at Teaple Dnxversity, 
San Francisco State Oniv^tsity, Michigan State ^Jnxversity. 
and UCIA. 

12 DeUlopers - Masters level personnel with at least one ye^ of 

experi^ce -m educational develojoent, or persons wxth Ph.D. s 
in education who have soiae association with educationa^develop- 
oent.or whb are presently developing educational products. 
Seven of this group were enrolled in inservice tramipg xn DDSE. 

4 senior Developers- - Highly e^rienced in developing educa- 

tional products an d prograins. These personnel f ^P^^^^^ 
worlcing in the field. They have published in the fxeld and/or 
pr3i2d products that have been distributed and are bexng used. 

78 TOTAL 



nfote analysis of var iance indicated that there were -° J^^^^.^J^p^''^^ 
between students and graduate students 50 -ihese grot^s were coabiii Jd _and are 
SSSSill^^den trin Table 2. Alt hough sW developers tended to have 
hSei ^t^I^T^ngs than developers the ^f/^i^^^^^^r 
toTsnallto demonstrate statistical significance. jBiese two groups were 
coifcined and are identified as developers in Table 2. . , 
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(a) the three context scales (i.e-, colum totals of figure S), (b) the siac 
proofigs skill scales {i.e.^ rov totals of Figure 5}, and (c) four e^irically 
derived cluster scales (see J^Ependix A . Please ^te that this groi«> of 73 ^ 
subjects is predoidnantly a student population, which is appropriate since the# 
Diagnostic Instruaent is intended for professional education settings. ^ ^e 
nurber of e35>erie2ced developers as sasall {H=16) but sufficient to deK>nstrate 
the ability of the IX)&B instruaent to discriminate 1bet5#een students and developers. 

iipendix A contains a .cCTplete report of the data analysis and a copy 
of the instruaent. !Ihe following pages suvarize the findings. 

2'- Content of the InstrTnent , • ■ j 

Analysis of the »data indicated that the DD6E Diagnostic Instruiient items 
were highly in^:ercorreldted hut that at least three or possilbly lour sAscales 
ian be differentiated. Cluster analysis resulted in four well-dtfined clusters, 
three associated vd.th the DDSS contexts and a fourth derived prijsarily from Itpms 
appearing in^e DSD contexts relating to publication, production; and public 
relations. However, correlation and^ factor analysis indicate that this last 
cluster is highly correlated vith and. eadiibits a factor pattei^i -siailar to the 
dissendnation cluster. Ohe factor analysis results suggest that only three factor 
scales are needed: (1) deveiopeent^ A2) field' test and evaluation, and -{3), pro- 
duction, dissemination, and oarketing. ^ . . - ' 

Neither the cluster analysis nor the factor analysis resulted in a-groi?) 
of conpetence statements associated primarily with any one of the process sJcillsj^ ' 
however, the s'jd.11 scale factor loadings on the newly defined EO^ factors eadiibit 
different patterns of skills loadings which af^ear to be ■eaningful.^ Ohe 
evaluation factor is most prominently associated with: ^^raUecting informtion, 
analyzing, and evaluating; the production; ^'sseniaaticfti an d mark^fcLng factor - 
withi planning and desiring, producing, and coa»jn|csti6g; the develogwent 

factor with: analyzing, ^planning and designing,/prodi^ing, and evaluating^ _ ^ 

■ ' . . X • 

The analysis, indicated that the it^ which.iiad bee)? placed jf pri^ori, in tl»^- 
developnent context, were in fact a.mixturc of all thr^EOfiE competencies, cpr- 
.•quently, tb^ nwber of Items associai;ed purely irith ^evelc$p«it is closer to 
nine In niMber- rather than the intoided -24. ' 
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TABLE 2 

DIAGTMSTIC lIBTRUKnrr ITEM KSMS t£SD SIGlJIFICaiSCE TES2S 



ABBROIATED TTQi COmEliT : 


mus 

5TUDEIITS 


KE«iS 
DEVELOPERS 


F- 

PATIO 


P- 
LEVa 


. DEVaO^iEMT 


tH=62) 


(N=16> 






1. Obtain infonaation on pn*lens 
.2. Locate strategies for deVelq?aent 

3. Prepare search st^-ategy 

4. Evaluate infonnation^jor reievcnce 


3.97 
3.73 ' 

4- 42 

5- 35 


5,62 
5-44 
■ 5-88 
6.44 


?-49 
11.88 
8.50 
7.55 


-003 
-001 
.005 
.008 


5. Write a 20 page proposal 

6. Use instructional theories in design \ - 

7. Review fieid test data for revision 

8. Cla^siiy instructional oujcci^ive^ 


4.05 

-3.87 

4.03 / 


5.00 

- -5^. 75 
4.94 
5.50^ 


5.04 
16-73 
8.86^ 
6.59 


.026 
.000 * 
.004 
.012 


9. Plan a budget 
10. Specify fonaat of materials 
IJ. Sequence teaming activities 

Cr4"tff)a'f'a AoorioH nmrltfCf"! DH daterislS 


3.50* 
3.68 
4.37 
3.81 


4.56 
■ 5.^1 
5.19 
5-69 


4.02 
9.40. 
2.70 
. 11.59 


.046 
.003 
.101 HS 
.001 


13. Secure waivers, releases, etc. 

14. Confer with specialists when needed 

15. Write instructional exercises . , " ^ 

^16. Write copy frotn specifications ' 


2.58. 
3.92- 
4.53 
3.76 


3.94 
, 5.19 
5'-56 
4.75 


-6.69 
4.90 
3.96 
3.04 


.on 

.028 — 
.048 
.081 "ilS 


17. Hake recotiraendatiSns from fj^d data 
•18- Conduct a cas'6 study of a^program 
19- eheck/product against specifications 
20. Inforaally try out a, product.' 


3.85 
3.95 
3.75 
4.55 


5.50 
o.uu 
.5.38 ' 
6.19 . 


13.37 

4 

10.31 
12.03 


\ooi 

.002' 
.001 


■■- 1 ' 

.21. Write a position paper 

22. Discuss a product with user groups 

23. Interact in a staff meeting , 

24. Communicate product specifications 

TOTAL - DEVElOPHHn" ■ " ' 


3.92 
4.03 

5.21 
3-.58 

3.99 


5.00 
. 5.62 

6.38 
5.62 

5.39 


4.27, 
.9.90 

8.41 
* 15.14 

20.12 


.040 - 
, ^003 ~ 

.005 
.000 

;4)001 



* - All Values tabled .000 have p-Levels of .0005 or less — 
Hi - Not significant at 5% level ' ^ 



Ti5SLE,2 (cont'd.) . . 



ABBREVIATEMTEH COflTEHT . 


HEANS 
STUKHTS - 


KEANS 
DEVELCPcRS 


J 

RAIJJU 


P- 


FIELD TEST i EVALUATIOR 


(H^2) 








25. Prepare a coding scheae > _ ^" 

26. Oi^anize sta^tistical data 

27^ Interpret scatter plpts v ^ 
IZ. Insure privacy in ^ta. coTlectioa 


\ 

~2.94 
4.31 
3.65 

13.76 


• 

4:iJ9 

5.62 
4.81 

5:25 


4.45 
7.05 
3.98 
9.10 


,036 . 
.009 

\047 . 
.004 


29. Discuss internal & external validity 

30. Fonoulate evaluation questions 

31 . Analyze test outccsroeS' 

32. Decide if tests fit evalQation plan 


4.10 

3.98 
3,85 . 
3.42 


5-06 ^. 
5.19 
- 5.50 
5.31 


4.17 
7.63 
13..94 
21.05 


.fi42: 
.007 
,001 
.000 


33. Detenaine validity of your^test 

34. Design^ a project monitoring systejp — 

35. Plan control of extraneous variables 

36. Set criteria for field test sites 


3.13 
'"2.56/ 
3.92 
3.37 . 


4,56 
4.88 ' 
4.69 , 
4.94 - 


10.98- 
21.96 

x2.62 . 
9.99 


.002 
.000 • 
.106*KS 
.003 - . 


37. Use evaluation data to revise tests 

38. Adjust test procedures when needed 

39. Prepare a test a^ia>stration mariual 

40. Discuss standardized interviews 


3.95 
3.48 _ 
-3.15 ' 
4.44. 


5.69 
* '5.06 
5.Q6 
5.62 


16.16 
9.20 
1^.82 
• 6.42 


.000 - 
.004-. 
.000^ 
.013 \ 


41. Decide if stati^ics are suitable • s 

42. EvalCiate test fnstninents 

43. Find internal consistency of a. test ^ 

44. Identify an evaluation's purjjose 


3.50 
3.60- 
3.56 
. 3.76 


4.44 

,5*56 
"4.69 
5,62 


3.91 
16.04 

5.63 
-14.96 


.04? 
.000 
.019 

.you 


45. - Prepare an article for publication . > 

46. Give a short speech or oral report 

47. pJ-epars simple evaluation reports 

48. Prepare graphs to display data 


4.15 
6.02 
• 4.87 
5.08 


5.06 
6.38 
6.25 
5.88 


3.35 
1.24 
11.21 
2.77 


.068 HS 
«269 NS. 
,002 
.096 Nf 


TOTAL - EVALUATION 


3.86 


5.22- 


19.59 


.0001 



* - All . Values tabled .000 have p-Leve'ls of ,0005 or ..less* '. 
NS - Not significant at 5j; lev^l 
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TASLE 2 (cont'd.) 





mas 

SPJDENTS 
U{=62) 


[EyaopERs 

(K=16) 


F- 

RATIO 


P- 


JISSEHKIATION & KARKETIHS ^ 

49. collect data pR^Tnstallation costs 
50- Desi^ narieting study data fonns 
51 ♦ Use inforsatlon services 
52. Cepstnict an aiinota:£d bibliography 


2.95 
2.79 
3.69 
4.55 


4.25 
3.69 
5.81 
5.12 


5.74 
3.75 
16.22 
1.05 


.018 

.054 ns 

.000 
-310 HS 


53. Idefitify rfiaracter of target group 
54^ Evaluate uarket research tediniqaes ' 

55. Deters^sne probissns inr-mstai laincn 

56. Secure cppyrig^-t clearances 


3.76 
2.34 
3.08 
2.65 


4.62 
3.69 
5.00 
3.75 


2.76 
. 8.17 
15.18 

4.05 1 


.097 HS 
.006 
.000. 
.045 


57. List isportant d^ssenination factors 
581 Design a public relations activity.-. 

59. Detename a laarxet tor your prouuci. 

60. Prepare specifications for A/y~Saterial 


' 3.53 
2.71 
3.34 
3.02— 

1 


4.25 
3.50 
4.31 
•3.75 


1-71 1 
I 2.18 
3.25 
2.12 


.191 HS 
.140 HS 
.072 lis 
; .146 HS 


61. Krite scripts for ^'-Ha i slide shovfs 

62. Hrite press releases about a product 

if t. 

63. Prepare a sfflnpie proaucc 

64. Write specifications for user manual 


1 3.19 
3.16 
3.56 
3.74 


3.94 
3.69 ; 
5.06 
4.69 


1.84 
0.95 

' 9.34. 
3.28 


.176 HS 
i .656 HS 
1 .003 
1 .070 HS 


65. Evaluate the product's distribution 

66. Evaluate effect of a deaonstration 

67. Identify parts of a narketing stu^y 
68- Interview users about product use 


2.77 
3.77 
3.37 
3.89 


3.75 
4.69 
4.06 
,5.12 


3.77 
3.80 
1.55 
5.97 


.053 HS 
.052 HS 
.214 HS 
^16 


69. Establish cooperation vrith tisers 
79. Deliver an oral presentation 

71. DeniMistrate the use of a product 

72. Translate data into verbal form 

TOTAL DISSEHHIATIO?! . 


4.15 
4.11 
4.53 
3.81 

3.44 


5.50 
5.38 
5.19 
4.38 

4.47 


7.34 
5.68 
1.64 
1 .06 

-8.41 


.003 
.020 
,202 HS 
- .308 HS 

1 .00051 



* - All Values tabled .000 have p-Levels of .0005 or less ^ 
tiS • Hot significant ?t 5% level 
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3- Discrini-natioa Valiaity . 

Given the predEcdnaotly st»aent oppulatSon of the field test saiole (62 
of 78 ST^jjects) , the average conoetence level and the distriijutioa of iten reans 
r€s>orted in ^ie 3 seess acceptable, since the average proficiency level for 
the entire instrtsaent is at the sai^int (4.0) on the seven-point scale, and the 
great isagority of the itea proficiency levels ar? between 3.0 and 5.0 (61 of the 
72 xtaiy. analysis of variance between the s. ,-ant and the developer grorros 
indicates that the sajority of the items and all of the scales discrianiaate 
between these groi^- In terns of the proficiency scale, the average difference 
is 1.27 proficiency levels, reans for the riajority of the ccsi^tence »-tate- 

sents for the stiident grorro were below 4.o/«hi<3i corresponds to the stateaent: 

-I have a general imderstanding of this activitj^ and[ have 
had scsae experience with it, enough so 1 can So ^.t^if 1 have 
either detailed instructions or close supervision.* 

On the other hand, the mjority of the oxroetence stateiaent zieans for developers 
were above 5:0, vhidi corresponds to the stateiaent; 

-I have enough eajperience in perfocaing this activity to 
do it if given enough general supervision or general 
instructions. " 

Hhis clear, meaningful, and highly significant difference (p < .0001) provides 
further evidence that the instrument is able to discrininate between groups with 
different amounts of relevant e«>erience. (Previous pilot studies Have shown 
that self ratings rade by educational developers correlate with supervisors- 
ratings of the cane sets of c«i5>etencies, -73 for one scale and .88 for another, _ 
thus indicating that e^rienced developers and their sts>ervisors exhibit sub- 
stantial agreeiaent regarding the average levels of proficiency.) 

li --terms of-levels" of self-rated proficiency, both the students and the 
developers reported relatively higher levels of proficiency in develqpiaent, 
followed closely by field test and evaluation, and relatively lower levels of 
proficiency in publication, production, and public relations competencies. Both 
gro,5>s rated corounication skills hi-ghest emd planning and designing skills lowest. 
The difference between the two groups was at least one prpficiency level for all 
subsc-xles (contexts, skills, cluster analysis scales) , with the greatest difr 
ferences appearing on the development context scal^, ±he field test and evaluation 



ERIC 



clisster scale and on thg aj^yziag, the collecting infezsation, and the eval- 
,|ating skills scales. She siaallest differences bet-een the two giOT^ vere oa 
the aissezmoatioa and marketing context scale, the (related) pifijlication, pro- 
duction, and public relations cluster scale, and the «»nmicStion skills scale, 
la the case of co™ication, the relatively saall diferecce in proficiency level 
seers to be atttihutable primarily 'to a (relatively) high level of proficicijcy in 
the student grots' «i1i respect to oral and- written coBaE«ication. in the case of 
the disssmnation and ssarketing context scale and the publication, production, 
snd carketing cltirter scale the sraller^ifference between gxxn^s is i^issrily 
attributable to the relatively lower levels of proficiency reported by the - 
developers. ' - 

4. Revisions ' * 

Given ttee results of the ^ta analysis, it was decided to retain the content 
structure of the instrument (asatxix of three contexts by six process skills), but 
to revise a nu=ber of itens so thac the three iX>SB factors vould be =ore clearly 
and evenly represented and r«re closely associated with the three context colmmas 
of the response fom. In addition, an eighth level vas added to the proficiency 
rating scale to separate the e^^^ert from the highly experienced, appendix A 
includes copies of both the original and the revised instrur.ant. 

2. IHTEBPBEiarnbN ailD DSE of mS-PIftCTOST IC ikstrdkeht 
f jll STDDEHT PROGRaM PZaiCilKG 

several field tests of various versions of the ISfiE Diagnostic lnstr«»ent 
have indicated that the instruinent is capable of discriminating between ine:^r- 
ienced students and e«>erienced DDSE personnel. Moreover, in those cases vhere 
we have been able to obtain supervisor's ratings for employed E06E professionals, 
self ratings and^i^ii^^tings have" di^layed substantial correlations 
{e.g. , .73 and .88). Bie individual itens and the subscales display substantial 
Intercorrelations , but factor analysis, and cluster analysis indicate that at 
least three different factors (or clusters) can be identified, corresponding to 
tte three DDSE contexts. Neither the cluster analysis nor the factor analyses 
provide evidence for separate skills fectors or clusters; however, the factor 
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analysis indicates that the three £Cffl factors (Factors J, II and HI respectively 
in Table 33 display loadings with different patterns of skills for I2MS- 

3he data in Table* 2 indicate that the average stodent tends to rate sost 
itess ""J" {^>od understanding bat have not yet done it) or "4* (can do it with 
derailed instructixms oi close si5>ervision) . Sowever these data, incltade a sub- 
stantial nuniber of students who had co=$>leted or were enrolled in educational 
BiD iboisrses. Entering, students snay be eaa>ected to display ^caeirhat lo»#er 
average ratings. Obviously^ oxzr best advice is for the instructor to cctcpile 
rating data based on students actually entering his or her own program and to 
use these data in evaluating the DBSE Diagnostic Instxuaent Eatings of individual 
students. - ^ - 

!ZSie instructor should keep in suind, and ej^lain to the student^ that 
the 72 itesss contain^ in the m>£B Diagnostic Instruaent are but saspl^ of 
a mich larger potential population of BDSS cosspetence items. Hence^ craution 
should be exercised -tn placing too much izz^^iasis on an individual iteia, or 
even on the average of the set of four iteas appearing in ^ch "cell" of Fig- 
are 5 in teras of making inferences about the student's conpetence; however, 
the totals (or averages) for colcnns (based on 24 iteos) and rows (based on 12 
xtens) are probably reliable indicators of the sttident's general level of com- 
petence in the specified area. 

regarding the validity of the self rating data, O^ble 2 clearly indicates 
that we can make useful and meaningful discriininations between groups of persons 
with different levels of e35>erience. Needless to say, individuals will vary 
in terns of their tendency to under- or overrate their coarpetence, and quite 
possibly in their interpretation of the scale descriptions and the laeaning of 
the conzpetence items thesaselves. Hence, it is essential that the instructor 
consider both prror and concurrent information about the stt^ent and about the 
student's peers in assessing and evaluating the Diagnostic Instrument ratings^ 
as inforaation about attained conpetence levels. 

pjs its label implies, the instrxment is intended for diagnostic purposes. 
It provides a useful device for reviewing and discussijig w;ith the student not^ 
only the student's perception St current levels of conpetence, but the student's 
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iaeas about^be levels of cans«tence wMch d»ald be set as objectives at 
the'cracpletion of the sttaSent's progran of instinsction. 

in terrg of cbiekives for st ^v^^nt proaraa olaimicg. the iten seans and 
totals reported in Table 2 for developers provide soae guidance. With s«e 
exceptions, it vbuld seer, that a -4- mating (I have a general tmderstanding 
of this activity .^a have had so=e e=a«rience vith it, enons5» so I can do it 
if I have either detailed iastructions or close supervision) or a "5" ratixjg 
CI have enough e^eince in performng this activity to do it if given enough 
<,eneral supervision or <;eneral i^nstructions) vpald be a reasonable objective 
(in an "entry level" i.e. H.A. degree program) fcr «.st, but i^t all, competence 
ite»s. Certa'inly the data in Table 2 suggest that the entry level student - 

should ai3a for rating averages of at least -5.0" i^ i^evelopaant and in 
^t and Evaluation. But note that €be developers, as a group, rated the=seives 
belo-^ -5- on 16 of 24 iters in Dissemnation and Karteting ccnts^. Cespite- 
this empirical self-report evioence, indicating that experienced developers 
average only "4.5- in this context area, an iv^rage of "5.0- ^ be a desirabXe 
dissemnation id r=ar3ceting ca=5«ten.^ objective for any student ai=ix.g for all 
round general cxzspetence in DDfi2. 

needless to say, students sho^d be encouraged to define and justify their 
>own particular sets of ca=s>etence c^jectives, not only in t«»s of the sample 
of ±t^ provided by the DDSE Diagnostic in=tr»ent, but perb^ by wrxt^g^ . 
their o-n statenents of co:5»etenciee they wish to- obtain. In ter^ of levels 
of proficiency, there laay be specific termnal. cc»petence ^teas at levels "3' 
(Studied but have not done it), -4" (can do vith detailed instructions or c^ 
supervision), "5" (can do with general s^rvision or instructions), and "6' 
can perform satisfactorily without s^rvision or jc* aids). Perhaps i^ s£«cific 
instances, there inay even be itei«s which could be justified at levels "1' (no 
Jcnowledge) or "2" (don't really understand)! ^ 

Ohe important thing in a cc»3>etence based, individualized program of 
instruction, is that the student and the instructor develop so«r tentative. 
but^xplicit, conception about both the individual student's er^ lev el and 
the desirable exit level for that^tudent. - l^e «gnitudes d£ the discrepancxes 
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between these tim levels loz differ^! sets of cxaospetencies can th^ be lased to 
help define the areas ia which hnowl^ge or skill attaiiaaeift is ne^ed. 

If not taken too literally, the fom displayed in Figure 5 can be used 
as a device that can be esployed to record the student's prelisinary ideas 
about tersainal objtetives and also to identic the isagnitude of the discrepancy 
between current and desired competence levels. Shis can be accoziplished by 
entering an exit level objective e»ressed as a rating scale nu22>er^ beside each 
average and then cQM)uting the differences between the recorded average and the 
student's objective. 

Figure 6 gives an (real) e^^imle for a particular student. In this case 
the objective (Cbj.) for eacA. cell (context by skiJrl coi:£>ination) is indicated 
iraaediately to the ri^t of the Average and is followed by the discrepancy CD) 
between the self-rating average and the objective. Most values have been 
rounded to one deciiaal. Kajor discrepancies are circled. This particular 
student is relatively low on self-rating for analyzing skills (2.8)^ planning 
skills (2.5), and evaluating skills (2.7) with several of the co25>etence iteas 
rated "l." In tenas of the three DD&E contexts, field test and evaluation is 
highest (3.8), followed by development (3.4), and then dissendnation and laark- 
eting (2.5). When conroared to the student item seans r^rted in Table 2, we 
see that the student's average for field test and evaluation is quite close to 
that reported in Table 2, the student's average for developzaent is about .6 
point lower, but the student's average for dissemination and saarketing is nearly 
a fun point lower, which is probably a significant difference. 
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-niis student displays considerable varianca in the rating of individual 
iteas with fs^itens rated -1,- and 11 iteins rated -6.-* -Zhe list of high 
and low it^ is interesting. Iteas rated "5- or -6- include (reading across 
rows} z 

(Collecting Infomation) 

4- Evaluate inforiaation foi relevance 

26. Organize statistical data 

27. Interpret scatter plors 

28. Insure privacy in data collection 
52. Construct an annotated bibliography 

(^alyzing) 

5. Write a 20-page proposal 
: 6. Ose instructional theories in design ,^ 
31. Analyze test outcones 

(Planning) . - 

11. Sequence learning activities 

35. Plan control of extraneous variables ' ^ 

(Producing) 

14. Confer with specialists >?hen needed 

15. Write instruct iojial exercises 
40. Discuss standardized interviews 

61. Write scripts for film and slide shows 

(Evaluating) 

[none] 

(Ctommnicating) 

21. Write a pposition paper 
45. Prepare an article for publication . 
4S. Give a short speec^i or oral report 
" 4*^^ Prepare siinple ^valuation reports 
4^. Prepare graphs to display data 

71. Denonstrate the use of a product 

72. Translate- data into verbal -form 

Apparently this particulai: student possesses significant skills in writing 
and comniinication, in some statistical/data processing areas^ and in several 
cojnpetencies deaUng with analysis, planning, and production of instructional 
materials. 

*The ratina variance for this student is considerably greater th^ that "^^^^ 
encountered, but not entirely unusual.- We have deliberately selected a student 
SSSwi^h substantial rating variance in order to illustrate the interpretation 
of the Diagnostic Instrument data. f 
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Ccaiversely -there are a nurf>er of ooapetencies in irhi« the student 
claiias no specific kaovJcdge ("1-) or understanding (-2-), including^ 

(Collecting Znforaation) 

25. Prepare a coding scheae 

49. Collect data on installation costs 

50. Design ssarlceting study data for»s 

51. Use inforsMition services', 

(Analyzing) ^ 

7. Beview field test data for revision lof a prodact])- 
30. Foanilate evaiuation _goestions 
32. Xtecide if tests fit evaluation plan 
* , 53. Identify ciharacter of target groijp 

54. Evaluate inarkeiL research techniques 

55. Y Detennine probleas in installation 

56. Secure, copyright clearance^ 

(Planning) 

9. Plan a budget ^ 
- 10. specif format of aaaterials 

34. Design a project monitoring systea 
36. Set criteria for field test sites 

57. List icgSortant dissemination factors 

58. Design a public relations Activity 

59. pe'teraune 3 inarket fo^ your product 

(Producing) 
,13. Secure MaLvers, released; etc. 

16. Write copy from specifications _ * ^ 

63. Prepare a sample product " . 

64. Write specifications for users' manual . 

• , (Evaluating) 

19. Check product against specifications 
42. Evaluate test instruments 

44. Identify an evaluation's purpose ^ 

65. Evaluate the product's distribution 

66. Evaluate effect of a demonstration 

(Cdnnunicating) 

22- ^iscuss a. product with user groi5>s 
23I Interact in a staff" aeeting 
. 69. Establish cooperation with users 

This appears to be a substantial nuaber'of oorpetencics, spre»d among 
all three DDfi& contexts and all six skill areas, where the student clai«i to 
have neither jcnowledge nor skills • 
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IJow, let us cake the very inportant assteption that the ix»structor has 
discussed these ratings with the student and has dooe enough probing of past 
training and experience to bfi willing to accept (at least tentatively) the 
ratings as reported in Figure 6.. Givei^ the stt^ent's soaewhat lower than average 
ratings in several cospetence areas, the student was encouraged to set objectives 
that would tend to bring rost of the conpetence rating averages more in line , 
aild closer to an- overall "5.0" level, 'which 'would indicate ability to perfbr- 
rost tasks if given general st35)ervision or general instructions-* 

'■>^'' ' . - 

(^jectives were actually set by considering the four "sarrole- items appear^ 

ing in each cell (ccHitLct and skills) . , To en^hasize the point that these four 
iteias are sanples, the instructor should .urge the student to nane another example 
of a conpetence that would fit in ^he saiae cell.. After the student has done . 
this for a few cells, the point is Visually aade satisfactorily- ^This step is 
icportant because we want the student to -set objectives in teraS .of a type of 
co^tence (e-g. . collecting inforaation for disseninat^on and narketing,) rather 
than only in tenas of just four specific item?. (e.g., »r 49, 50, 51 and 52). - 
After working with the student to think JthrougK gig^task o f settin g objectives 
and ooir?»uting'the discrepancies between current averages ^d objectives for a 
•few cells, students nay be left to conrolete the task on their own. ,The instruc- 
tor should review the coiii»leted form with each student in order to determine the 
student's understanding of and reasoning for the objectives that have been set. 
Obviously, if there is evidence of unrealistic e35>ectations, the student should; 
be encouraged to nodify the objectives to bring them more in line^with what the 
student can attain. 

In terms of the example given in Figure 6, this student initially aimed 
for 5.0 averages for "the objectives ^in all three DDSE contexts. However, after 
considering the veiry low cinrrent average in Dissemination and Marketing, several 
cells in this column were reevaluated, and the instructor and student mutually 
agreed on objectives with a slightly lower average in this context area (4.8\. 

♦Actually, this is a fairly ambitious overall objective. In aany cases an 
average closer to 4.5 may be more reasonable. Because this student had some 
prior RfiD work experience, had already attained "5" and "6" ratings in a 
nrnJaer of items, and strongly wanted to work in the edubational RSD field; 
the objectives appeared attainable. - ' 
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in term of skill axUs the stuaent especially wanted to wofk on planning- 
skills, so d^ite a current average rating of only 2.5, tie s^<Sent set 
objectives in this row wiiich called for »ajor attaiiment -ir all three DDtE 
cells, ihe areas wiXiargest discrepancies faetyeen current and desired cot*- . 
tence le..els are cirJled. Kajor discrepancies exist in .11 of the Dis.e.ination 
and^Karketing cells and in all of the glannipa: cells. In '.dditlop tiiere ^ - - 
large discrepancy in tOxe cell educating in the ei>ritext of T^o^t. ^ 
Perhaps- notable is the fact that t«o cells sl^oir zero^ di.crepinci^S^d«cina and 
comunicating in the ^ntext of ^.-qI^ T-..t and Evaluation. Overall, these p^^i- 
cular discrepancies piint t?b the need for a. fairly co»erehensive progra. in all 
facets of -KJSE: but clearly with the need^ for practical work experience of so« 
kind to accompany the acade»ic phase in order tp acquis;* practice in a wide 
ra^ge of DDfiE skills, and with substantial effort directed, to aoqoiring co-pe-, 
tencies in Pi.semination and Kariceti na context and in the Plannina *nd the 
Analyzing skills- areas. 
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' III. PROGRAM PROGRESS jFfiSESSHEOT 

We agsuse that the studisnt and instructor will develop a tentative plan 
for a prograia of^'ins traction based on the assessiaent and planning at prograa 
entry. If the progtan is truly cosipetence-based/ there should be provisions for 
credxt by^gxaiaination, and advancement ^hoiild be based on deiaonstratioh of attained 
cosoetenbe rather tnan siinoly csodules read or courses taken. All of the OTfiE 
instructional nodules soecify instructional objectives, and the majority of them 
contain *end-of-codule written examinations. . Hence, the DDSE instructional mate- - 
rials can provide scce assistance in jaakxng program progress assessment; hcrrfever, 
this information will be primarily in terms, of the student's knowledge of tech- 
nical terms, concepts, and principles. Where'veiT possible, the instructor should 
attempt to add information about actual performance (in terms of instructor, peer 
or work supervisor's ratings, observed performance in laboratory, intern, or 
on-the-job settings, and in tenss of products, such as plans, outlines, analyses, 
prototype products, evaluation reports, etc., which the student produces). Our 
field tests of various versfons of an assessment. battery consisting of (a) self- 
ratings, (fa) knowledge tes'ts, and (c) perfcmance test siskilations, which are 
^discussed irr.OTction VJ, indicate th'at «hile eaA of these t^ree sources of infor- ^ 
'mation Jiave srane validity, they are far from redundant. Moreover, knowledge 
tests appear to be the weaJcest of the three sources of infoniation, at least in 
terms of discriminating between students and professionals. |vppraisals of students' 
products and ratings of specific conroetence areas made? by thc5^e who haVe been able " * 
*to observe the student's behavior in work situations (e.g., instructor, peers, or 
work supervisors in laboratory or real work' settings) can add. important information, 
which should be considered in assessing competence levels. 

1 * 

- We are able to offer the instructor only modest ^sistance ?.n this regard. 
With respect to the. appraisal of products, we atte^^^d failed, to produce - 
a "general" product rating fom t^-t could be empl(^ed «^&>^variety of products. - 
However,' we have been able to develop a scoring manual for the simulation tasks 
described, in section IV (see also Appendix E.). Examination of the methods of 
scoring used in this manual may provide ideas for scoring other products. Basically, 
we have att^ted to appraise each' product in terms of four types of questions: 

1. ^ .(Pass/Fail) Is the product acceptable?, ihis 'is a p^s/fail deter- 
^miijakon which mst be based on the instructor's judgement. Jn sane 



. instances the instructor may have sufficien^^ea^rxence to be 
■ imite sure about the judgment. In aany cases,- at aay be illum- 
nating to ask a few RSD professionals who aqtualiy s^rvise 
entry-level professionals to examne^a sarnie pf student pr^cts 
and inake their own appraisals in tenns of ;wfiether they would 
accept the oroduct or not, and why. Our own 

with critiques of the simulation tasks we have developed ^^^^^ 
that this Sh be* a highly useful kind of '^^^^Jl'^^^'r^^l^Jf' 
ray cause the instructor to refine or. even significantly «odify _ 
♦ the criteria for pass/fail. . , 

2. (Fractional Pass) . How close did the product co«e to passing^ and 
what is unacceptable about it? If the product is not J^cept^.e, 
Lswers to th^e questions are essential if the student i^to 
obtain , feea^ack-^n-perfon^ce and take appropriate ^^^e'^Tf - 
action.' aqjhasis on what was done properly can provide positive 
reinforcement for study and work acco^lished specification of 
precisely what aspects reraire further attention .gives the sttident 
needed guidance on how tc produce a better product. . . 

3. (S^pe of Application) In what way' does this prpduct, J^g^efes 

of Sss/fail on acceptability, reflect ^PPji^^J^". f ^^^^ 
knowledge, skill or sensitivity that goes beyond t^e pirtit^ilar 
SSS^ r^qUireinents for acceptability? We have atter,>ted to . 
TJ^e proScts for -scope" or "breadth" of ^I^i-ation because 
.we hav; encountered substantial variance in the ov?Srall quality 
' of the products produced in response to the specific require 
oents. Criterion referenced testing is often presented as. a 
^Svfail". our second appraisal question addresses the probl*-s 
o?^.'grce.. of failure. ' -Iliis question addresses ..another pr^^^. 
. ' °t oSrall quality of the perfornance. ^ F^edbabk to the student 
•»«re not JTterL of what i«ust be- corrected to produce a 
passable product, but rath^ it is on i^^ntification^of and 
poSSve ?Snfprd«nenffor any outstanding feature of the product. 

' 4. (skeWic Co:.^nent scores) What can be said about t|e 

specific aspects of this particular t2Ee of product? , ^ 

. pg^ious three questions -reflect a global appraisal of^ overall 
- products, ^st products can be divided into several - 
Sbich-can be apprasied individually (e.g., quality o£w^t^ 
evidence of use of appropriate references, f?"f°^"2j5^tn the 
specifications). Our scoring in -this, situation is related |o the 
_ _ -afractional pass^ score discussed under #2 above; ^"^1*/°'.^ 
■ ■ tJf sake of scZing economy we have not atte,|pted to «ake * close 
link The "ftactional" assessment deals exclusively «i« 
L^;:?;; IZJ^t^ Of ele^nents or characteristics ^-<^.^^<-^^^^ 
be considered in e stablishing " . P^--/^^i,^^*^^^?:V,^, ^f^" 
fic coii5>onents" assessment selects only the icst i.portant p | 
^ior elements of a particular kind of i«roduct a:^ singles theH 
otit.fbr special attention. *j ^ • 

Now these are quite general guidelit^es. Ohe reader should refet to 
Appendix 2 for' exan^les of their application irf specific instances. 



Hith respect' to perfornance "ratings, ve' have eaployed the sisple device 
of dJanging the person reference of oar rating scale (see p.l9 ) froa "1" to 
-She/fee,- and then asked worh supervisors, student peers, voxSi colleagues, or 
instructors vho vere kno-^ledgeable of the student's perfeiaaance to rate the 
student using iteas froa the DDPS Jlssessaent Instrujaent or the si^^pleaentary 
iten pool. i?aters should be enco.i2saged to rate the student on all co=E>etcncies 
where they have enou^ information to feel qualified to Bak<s a rating , but 
should be allowed to ordt coapetence iteins vhere they feel that they lack relevant 
information. (Sn ircortant discovery for us ia this regard was the fact that 
mny inaediare si5>er^sors of MifiE ^fessionals were unable to rate the pe^^oas 
they supervise on many of the diagnostic items. In soae cases this was because 
the particular kind of ork had never been perfonned under their supervision, 
but in many cases it was because they singly did not supervise closely enough 
to have any good is^ression of the enpjbyee's capability.) 

On a more general note, performance ratings,.'i€ supplied by others, and 
Eost certainly if supplied by peers, should be attei^rtied only in a -formative 
evaluation- context and with provision fcr full and open ^<3:anse between rater 
and ratee- Under these conditions, the ratings can serve to establi^ positive 
and beneficial corsmnication. Anonymous ratings by others can be highly useful 
in one-shot, go/nc-go assessment, but are not reco=nended in this instance, 
because the student needs to "verify" the basis for the rating, and may need 
help in doing something about it if it is below his/her e:q)ectation. / 

in section II we outlined the type of file which needs to be kept on student 
■ progress. In that section we also introduced the DDSE Diagnostic Instrument 
Record Form, and illustrated how it could be used .as a device to initiate program 
planning by recording (a) entry level coii?>etence ratings, (b) program objectives - 
ratings, and (O the discrepancies between these "two ratings for each ceU in 
Figure 6. It seems to us that a simplified extension of the device could be 
used to provide a summary of the student's accor5>lishaieht in iach area. Cne 
possibility would be to use a simpler matrix form (such as Figure 4) and enter 
. in €ach empty cell of that matrix pertinent data repres^ting progressive estimates 
of competence levels, as well as revised statements of objectives. For a 
possible example of a series of cell entries see Figure 7. . 
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4.3 — 

4.4 — 
4.8 — 
5.2 ^ 


4.5 5.5 
5.0 — 


2.5 : 5.0 1 
3.5 ' — 
— 4.5 
3.P 4.2 
4.5(*.3} ^ 
- 


Attaljrzfng 


4.0 5.0 
4.2 — 
— 5.3 

5.0 — 

5.1 (-2) 


3.0 5.0 
3.5 — 
3.7 — 
4.2 4.5 
4.6K1) 


1.5 4.0 
3.0 — 

3.8 4.0 
4.0 -y^ 




i 


1 





line 1 — 10/12/76 
line 2 - 12/3/76 
line 3 ~ 1/17/77 
line 4 — 3/16/77 
line 5 - 5/11/77 



in each ceU assessed level of conpetence, as agreed upon by student and 
instructorr appears on the left and current objective acinars on the rig?at. Date 
of entries, of each line are recorded on the ix>tto« of the for.. D«hes indicate 
no change frc« previous agreed ^n levels for actual attainment or for objectives 
Check Barks (perhaps, in color) signal att^naent of the -objective. Entries 
sudi as (+.1) indicate where final level exceeded the objective. Note the fxnal 
entry in analyzing in develop^t is (-.2), indicating that the student fell 
short of attaining the objective in this area. 
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Such a sisnznary statsis ^eet on estinated cxmpetencies and objectives 
vould be »iatiyeiy easy to jaaintjain and can provide at a glance the overall 
picture of the stodent's proqress. Obviously the data contained in soch a fora 
can V3 but a niraerical "shorthand" fo- the icucA ricSier set of notes, records^ 
r^rts^ and other ssaterials which mst be contained in each student's file, 

As a final note^ our assusrption is that^ despite their brevity, the conpetence 
rating ievf Is illustrated in these sizmaary status sheet entries, will be based 
cai a 3oint assessnent aade by the student and instaructor, tajdjjg into account 
ail of the available evidence at each point in tiiae. Hence, these are not sijroly 
self ratings, or instructor ratings - but rather an overall assessaaent based on 
end of rodule test scores, performance on student projects, appraisal of reports 
and papers, and other kinds of evidence that ray bear on Ba}:ing a reasonable 
inference about the stuSent^s current level of proficiency in each laajor cocipetence 
area. 
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A. aSSESSKZHT C? ZXTT C0*gggaC3 

It seeas desirable for a c6i:5>etence-based training progrs. to have s(*e 
systematic reans fcr reviewing evidence of attainMj«»6«tence and for 
certifying student attainment at the tixe the prograa is ccapleted. ^ can 
be done by reviewing the entire student file and =aJ:ing a final set of 
appraisals, which ^uld incl^ not only a final set of proJ^ency xatings, 
bat sumary annotations, based on the instructor's notes and otiier doct«n- 
tation, regarding the character and quality of the student's perfer«ance. 
Ihis type of docunentation, vlll provide the student, the instructor and 
prospective einployers vith a useful fora of coipeten^e-based certification. 

Eo«ever, we felt there was a need for soae type of final exaaination, 
that ni^t be used at the end of an instructiooai series or at the end of 
the entire OT&E program- 

After pilot development and test of two -mni-batteries," each based on 
^ single DDSE instructional rodule, it vas apparent that the project would not 
have the t«ne and resources to develop and validate certification materials 
for the entire 0DSE program. We narrowed our efforts to the developaent of 
a Conpetence Assess=«nt Battery (CAB) thgt would certi^ skills in the area 
of -product developnent." The DDSE Training Program has five Develoj-ent 
series instructional rodules, which cover product design, production engineer- 
ing, review, tryout, and revision. OSie ^ve modules provide orientation 
and ^.^^^^.^-.-ation with the general pr-ocldures and problems eii5>loyed in 
product development. Because review, tryout, and revision are considered in 
the series, the instructional content of the series spans both the development 
and the ^.^.l^ t^t and evaluation contexts of the DDSE competence «trix 
(Figures 4 and 5). -»toreover, virtually all of the process skills can be en- 
co=5>assed. For these reasons, this series aH>eared to be the -ost pressing 
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loth in terms of validation of a Competence Assessaent Battery (CS3) -nofiel,- 
and in terrs of providing DDSS pi-'grsn :i »rs with a core set of Ca3 iteas.* 
■ae final version of the Developae.-. Series CX3 consists of three itesss 



• EISS2 Activities (a rating forsa) 

• Job Knowledge QoBstixxas (a short answer essay kncrwledge test) 

. • An Exercise in Educational Develocaent (a performance sii^ulation) 

' ♦ 

1, REDS2 Activities 

Ihis rating instrtsaent consists of 60 itens, sisailar in character to 
those appearing in the !»S2 Diagnostic Instruaent, but this tisap including 
coinpetence content dealing with research and statistics, planning and writing, 
and sanageiaent, as well as developzent, evaluation, and disses-iiiation-** 

-Sie RI»SE Activities iters can 35e self-rated Itj' the student. And xa 
sore cases the student nay be rated on the sare iteas by others (e.g., instructs 
or work supervisor). -The instriraent is self-adainistering and usually requires 
ten to fifteen minutes to complete. Ihe instrument provides scores on six 
10-itea scalesz (1) Research and Statistics, (2) Evaluation, t3) Instructional 
Product Developr«nt, (4) Planning and Technical Writing, (5) Dissenination, and 
(6) Rsa) Managenent. 



*ln oizr opinion, the next nost useful addition would be to develop CaB iters 
in the Dissemination and Marketing Context in order to provide sone coverage 
of all three contexts. 

** in developing the DD&E instructional naterials, we assumed that cost 
students would also take courses in. research and statistic^ to augment thexr 
work in DDSS. Although rarely taught as a specific course, the AERA data had 
sensitized us to the existence of RSD management co=^tencies. Hence, it 
seensed desirable to include a sampling of iteas zn these other content areas. 



59 



46 



validation field test version of the self-xating instru-ent e»s>loyea 
a si«5.1ified six level ^for^ce scale whi^. onits all' references tc tocledge 
ar^ focuses solely on perforx^ce.* ^e scale is as foll^. 



Level of gerfozaance 

1. I cannot perfcnn this activity even with si^rvision 
or guidance. 

2. I can perfoin this activity if I liave either detailed 
instruction or close supervision. 

3 I can Derfcm this activity if 1 aa given enoiigh 
' general stg>ervision or «?eneral instructions. 

4. I can perfona this activity quite satisfactorily 
without si3>ervision or job aids- 

5. I can-perfora this activity quickly and efficiently 
and can do a top quality job.. 

6. I consider nyself an'e^ert in this acti^^^^ 
can acco=5>iish unusually difficult or co3S»letely 
novel work. 



ERIC 



>Please co:^ar^ ^e™^ sc^ with^^^ ^i'jScy 
DDSE Diagnostic Instrun>ent (p.19) .- ^"Jt^e leve^ -^^ersSnciing 

r^^^'a;"ne XTLln f^LSliL^t^rSS-'ii'vS^n^ Of perfor^nce 
Xri I n^t ^Tfo" this -ivity — ^ -:^2^^SS^ievel 

proficiency ^^^^^'^T^^^^^ ™ ^ 

2; 5 was" added. A rough but probably satxsjctcry 

SSrS^SSn'STe sL^r^L'be ^de by adding two points to th. ^er^r^ance 
^:Tby Sictina two points fro. the proficiency; scale. 
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2. Job Knwledqe Qaestions 

7?»is short-answer essay Jtoowlefige test consists of tea iterss dealiiig prisaarily 
vitb instructional prodoct develcpoent and evaltiation. iMs test had a tiifi 
linit of one bora:. Itoe ten questions involved the folloang content.* r 



I 

(1) Identic* isrportant infornatioa mssing in a product 
description. 

(2) Specify appropriate tenainal objectives a briefly 
desc2ilt>ed prodact- 

(3) toalyze and ^scuss a proposed product tryout in 

terzs of the relevance and is^rtance of the aSiforoation 
it would yield- 

(4) List three of the basic criteria for judging the adequacy 
of revisions vhidi have been laade in [any] product. 

(5) Coapare the advantages and disadvanta^ of Sound oovies 
and souni f ilrstrips as instructional laedia ccasxdered 
froa the viewpoint of the develc3>er, the distributor, 
and the user- 

(6) Indicate three questions a developer night want to an^r 
in a tryout about student satisfaction with a prograixned 

text. . " 

(7) Identify the infbraation that could be obtained froa 
a consultaht'revie- of a product that not be 

- - - obtained Iji a .field t^t of the product. 

(8) List three questions (for each of three sp>ecified tM>es 
of reviefers) which should be aslced about a IspecxfxedJ 
product. 

(9) Identic factors other than the, product itself, which 
night account for difficulty subjects say encounter on 
using [anj3 p«>duct effectively. 

(10) Outline the sequence of cajor steps that are most i^roortant 
in the production of aisound filxastrip between the two 
stages (a) definition .of the instructional objectives , 
and (b) collection' of the first pilot test data. 

* » *0 

% 

*The content listed h ere is soaeti^s abbreviated - .""^"^ l^.""^^ 
sZlaTns the co=^>lete forn of each question c.-^ the scoring xnstructrons. ,^ 



Sac3i iten vas scored three ways: <a) pass-fail (P-?), scored X or 2; 
{bj frabticai (F) of essential elements covered in the response, scored 0 - 9| 
and (CI scope {SO of ajpUcation of kMMledge going beyond essential require- 
osnts, scored 1-5- 

3. An Eagrcise in Sducatidaal Developgaent 

Ihis is a self-adninistered sisalation test vM<^ recpsires approxiiaately four 
hours to comlete. Hie siasulatioa contains an introduction vhich explains liow 
iSie^ianilation is organized, what the student is ea^ected to do, defines the 
stxident's role as a zesJber of the sissalation's develoi»ent tea«, and provides 
bacJ;ground information regarding the product. Hhe tea. is working on a »anuai 
for- gaining librarians of a "toy library- how to teach parents to use educational 
toys with their .children. The introduction is followed by four sequentially 
organized tasks i 

Task 1. Prep aring Guidelines for Develop ers of the Librarian's 

Manual/ (1-1/4 hour tiiae allowedT This task requires the test 
subject to read several docuasents, extract relevant infor- 

ration, and organize it in a ie^o- so the deyelopoent teaa ■ 

will know for whoa the inanual *is. to be written, in what 
form it is to be prepared, what it is si5»posed to acoo^lxsl}, 
the specific steps the librarian should follow in conducting , 
a parent training session with the selected toy, etc. 

Task 2- Preparing -for the Tryout of. the R esource Kit. (1/2 hour) 
■Hie subject is required to do two things in tlur'taski 
(1) describe wnat the Researdi and Evaluation (RfiE) Tea» 
(which would conduct the field testing for the deve2op«>ent 
project) should look for in selecting test sites, xmrarians, 
and parents in order to conduct a field test of the librarians 
manual, and (2) list the questions (separately for librarians, 
parents, and RSF observers) that the subject %#ould like to 
have answered in the tryout. 

f^sk 3 Recaapendinq Revisions for the 'Librari an'^ Xanttal. 

a hour) 5n this task the sii>ject is presented with a five- 
• page sui«ary report of the tryout of th« librarian's mnxiMl 
at two test sties. On the basis of this s»ry the subject 
is asked to do tfeo things: first and «ost important, list . 
suggestions for revising the librarian's »anual based oo the 
*ryout results; second, critique the tryout and recoinwad 
what additional instructions should be given to the RSE tea. 
before undertaking fxirther tryouts. 



62. 

49 



Task 4. Outlining a Script. (1 hour) Biis is a fer laore specific 
task that would be appropriate only if the student has devel- 
oped ioise co2spetence in developaent of audio-visual instruc- 
tional isaterials. It: Ccdls for the subject to rough outline 
the content of a short sound filiastrip designed to provide 
a "rodel" for parents to view before attesting to ro3^-play 
"parent" and "child" iples in practicing to use the toy with 
their children- 2n addition the sxibject is asked to produce 
sosie detail (both audio and visual) ^or only the first several 
^ ^ fraaes. 

Appendix B reports on the developaent, pilot test and field test validation 
of the conpetence assessrient battery. Appendices C^D, and E contain the instru- 
nents and scoring instructions. 

B. RESDLIS OF THE VaUDATIQN TEST OF IHS BATTSRY 

1. Validation Test Subjects 

Sixty-six subjects were recruited f ron ten sites^ including four univer-; 
sities and six R&D agencies. In terms of experience the validation subjects 
33ay be divided into three grougs:^ 13 students who had craipleted relevant course 
work* but had no educational R6D work eatperience; 21 subjects (mainly students 
or recent graduates) with nodest work experience and some record of producing 
R&D publications or products; and 32 subjects with significant R&D work exper- 
ience and a substantial record of publications and products- Table Bl of 
Appendix B provides data on sex, average age^ teaching experience, R&D work 
experience, and numbers of publications and products for the three groups. 
Table 4 susnmarizes these data. 

/ 

There are substantial differences among the three groups in age, years of 
R&D work experience, nuinber of publications and number of products. Although 
the majority of all three groups have some tea'ching eogserience^ nearly all of 
the modest experience groi?) have such e35>erience, Note also that the high RSD 
experience group is preponderantly male, but that females are in the majority 

♦Typically, these students had completed one course in research design, one 
course in tests and measures, one course in instructional technology and one 
or two courses in statistics. About half had also completed a course in 
evaluation. 
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BIOGRftPHIC DATA <^ HBZ vaUDATICK GRDOPS 



•4 





RtD EXPEiaHKE MED PS£a»CTIVITy 




Ko 


Modest 


HiQfa 




•Kmfcer in groTS> 


- 13 


21 


32 




Age in years 


24-4 . - 


. 30.3 


38.4 




Percgnt aale 


31*' 


38% 


84% 




Percent who have taught 


62% — 


91% 


69% 




Years RSD vork eag^rience 


0-2 


1.4 


10.1 




Huafoer of publications* . 


0-3 


2-2 


17.;6 




Huaijer of products* 


0-2 


5-4 


21.3" 





*Hote that figures for pumber of publications and '^f^^^f^l^ff.J^ 
sums of averages for five separate categories ^^^^^^^^^/^^ S^dixB 
separate categories of. products which are reportedm Table B 1 of *pp«ndixB 



I 

% 

) 
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for the other two groups, -fte average individual in the aodest RSD ea>erience 
qrotip has worked in sor« type of educational aaD work setting for 1.4 years, 
has produced two publications and five products. Hie average individual in 
the high e«>erience group has worked ten years and produced over 17 publications 
and 21 products. 

2. Results- for the 5DD5E Activities Self-ratings 

All six scales of this self-rating instrui=ent were significant when t^ted 
by one way analysis- of variance based on scores of the three RSD e^rierce. 
level groups.;. Table 5 sursarizes the results. 

rhe student group with no RSD work experience' (but «ith sone relevant 
training) attl>ins average scale ratings which are close to or slightly above 
3.0 {-can perfonn if given enough general supervision or senerai instructions").* 
By contrast, the high RSD experience group has average scale ratings above 4.0 
(-1 can verfom this activity quite satisfactorily without supervision or job 
aids") on tour scales and above 3-7 on the other two (research fr statistics , 
dissemination). Hhe group with rodest RSD experience has scale average ratings 
that are intermediate on all six scales. 

It should be noted that the items appearing on the six self-rating scales 
have not bee. selected on the basis of group discrimination; but rather for their 
coverage .f RDDSE content that seemed appropriate for entry level professionals." 
Given the very large differences in work experience and productivity indicated 
in Table 4, the general character of the self-rating results reported in Table 5 
should be expected. For instance, note that the high Rsb exp.erience group has 
average ratings that are substantially higher than the other two groups on the 
RSD Management Scale and on the Dissemination .Scale. In these areas, the members 
of 'the modest experience group have not rated ^themselves appreciably higher 

*Sihce there are ten ite.n;s on e^ch scale, the kverages «P°f ^^^f^^?^^ 
b^ lead as performance scale ratings by shifting the decxmal one P^^^J^^^^^ 
lefr (i.e., the 30.2 average of the group with^OK, RSD ^ ^^^"^ 

'and Statistics 6'cale is 3.02 on the performance^ ratxng scale) - 

**Table B2 of Apoendix B presents group means afad P'^f ^^^f J^^^ 
variance tests of differences among the three g?oups for all 60 xtems. 
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RDDSE ACnVI'iX EATIHG SCRLS MEHJS 
>JiD SIGHIFICftSCE (P-ISVEL) OF F-TESTS FOR VM^DKnOH-GBCH^ 



1. Research and Statistics 

2. Evaluation 

3- Instructional Product 
Development 

4. -planning and Technical 

Writing 

5. Disseiaination 
6- RfiD Management 



MP EXPEgTTMCE UBVZI 



Ho 



30.2 
31-3 

33-2 

33-7 
31-2 
30-5 



Modest 



33- 2 

34- 3 

38-8 

37-3 
31-6 
31-4 



High.. 



38.3 
41.6 

41-5 

43-2 
37-2 
41-9 



.05 
-01 

-01 

.01 
-01 
.Dl 
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than the no e3q>erieace group. However, on the other four scales, the laodest - ^ 
-€35>erience group has average ratings that are aore 'nearly internsdiate, biit 
closer to the no eacperience sroup than -the high es^^rience groi^* in Research 
and Statistics and in Evaluation; and closer to t£ie high experience group in 
Instructional Product Developoent and in Technical %:xting- 

- - . < * -■ 

Although our focus in "validation -is ^mainly related to work ej5>erience 

criteria, we note that the self-rating scales are also significantly and dif- 
ferentially related courses taken. For instance, ^there a£e modest to substantial . 
coSelations (.41 to .63) between both the Research and Statistics and the Eval- 
uation scales and the nuiifcer of courses taken in Research" Cfesign, in Tests and 
vteasures, and in Statilti<^,'as well as in "other" (than educational) Psychology 
courses. Hence there is evidence that the self-ratings reflect the individual's 

level of training and 'work e35>erience. ^ , ~ 

— — < , 

scale' jneans, -standard deviations, ^pha reliability coefficients, and 
correlations are reported in Table 33 of Appendix B'. That table indicates 
that al3 six scales have internal consistency reliabilities of .90 or higher . 
and that the intercorrelations aiaong the scales range from .40 to .82. Olie 
Research and Statistics scale is strongly correlated (.79) with the Evaluation 
scale, but otherwise displays the lowest correlations with the reniaining four 
scales (.40 to .51). Instructional Product Development is the next most 
independent scale with correlations ranging from .51 (with Research and Stat.istics) 
to .74 (with Planning and Technical Writing). By contrast, the Evaluation sg^le 
shows moderately high correlations with all five other scales (ranging from .66 
to .79). The Management scale also displays noderately strong correlations with- 
in sc^es; the highest being .82 with Dissemination and the lowest being .51- 
with Research and Statistics* 

TO summarize, the field validation data indicate .that despite their brevity 
(ten Items) each of the six scales has high internal consistency, is able to 
•discrininate among subjects with different degrees of work experience, and . ^ 
tends to correlate with number of relevant courses taken. The size and range 
of the scale intercorrelations, (.40 to .82) indicates .thai^^ndi^dduals- ^end to 
rate th'emselves at roughly similar levels on alL -six scales, but that the scales, 
are' sufficiently independent to treat each as a separate scale. 

^ it 
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3. Besiilts for the Job iawtf ledge Ooestions 

Each of the ten essay questions ir this test were scored three wayss 
pass-fail {scored 1 or 2); fraction of essential .ele-ents covered {scoreS 
0 to 9); and scope of application of Jcnc^ledge going beyond essentia requixe- 
-aents-<scorei 1 to 5), Because ±he pass-fail tP-F) score is^qperitionally re- 
lated to the fraction {FJ score, it is not prising that the correlation - 
between these two scores, averaged oyer the ten it^, is .87, ^ =s«s« (SC) 
score, which «?y be considered a^-bonus- score for responses which repeal ^ 
"scope- of a^lication of Jcnowlelge going beyond the specific ^sential reqaire- 
laents set for each questixa displays substantially lower Average «>zxelations 
(.^8 Kith fraction aAd .44 with pass-fail) , indicating that the^^ ^c^re _ 
do^s contain infonoation ,nbt present in the pther scores. Al^P^ 
are significant intercorrelations aiDong' these *hree scbres -for each ^t^, ti* 
intercorrelations o'f scores- ^ng the ten questions were surprisixigly 
. with only -25 (signifioknt at P < -05) . A factor analysis of i:he ?p spres 
produced as W factors as there were questions- Apparency the ^cnqwfedge-, 
required W deal with these ten Job Knowledge Questions is -pderat^y specific 
to^-each item; however the guest^arenoot 1u>tally in^pendent: For ^ns^tance^ 
there are Verv =odest but si^f icant correlations, ancng the seyeral .gueatiqns.- 
deaCLing with review of tryout, and there is. a tendency for fraction or scope 
scores of' one question to be related to scope scores of other questions, botr : 
none of these relationships are particularly strong. , 

^^ysis of variance of score S^ences a«ng the three e^^ence 
;roups indicated that only four of the tf questions had ^:ccre ..ean ^fferences 
■ that were significant at the -05 level. Hc^^r Ihe total -scores .(s«ied . 
'over the 10 itesjs) are all significant. Se.e Table 6. 

Hot^ that the. modest MD e^rience -gr6i5> has siighky M^.toJ;«l/*cprep 
than tie high e^rience gro^ (all non-significant) . ?6th of ^ m^J^^ 
have significantly higher scores than the no .e3Q>erience grPH>. iJ5>r^.«ed 
as percent of the possible, score we see that appr6^€ely'l5.% jore of ^tije 
e>5,erienced subjects "pass; " hc«ever, the absolute percentages i«f^xng are 
troubling, primarily becat^e they indicate that even the «,re e^rience^ ^cups 
were averaging.only 6-9 and 6.7 questions .passed out of ten. Althpi^ ^ 
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F-HATIOS AiJD P-ISV2LS FOR IHE 
P-F, AKD SC TOTAL SC0H2S OU JOB KSGWL2DGS 2D2STIOSS 





: SSS) SXPSRIEKC2 S P-2SV2L 










Modest 


High 


F-Hatio 


P-Ifivel 


Pass-Fail (P-F) 
CP-F as % possible) 


15-3 
(53i) 


16-9 
(€9%) 


16-7 
(67%) 


3-81 


0-&27 


Fraction (F) 

(F as * possible) 


70.6 
(78%) 


79.5 
(88%) 


77-9 
(87%) 


4-89 


0-011 


Scope (SO 

(SC as * possible) 


25.2 
(50%) 


31-7 
(63%) 


29-8 
(60%) 


7.54 


0-002 
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fco«r t:i»e liz^t be partly the prc^l«. it see«s, in retrospect, that the 
oass-fail scoring standards (see D ) vera unreali^caUy hign, «xtb 

; fe- exceptions, a -pass- score is obtained only if the s^Ject's re^ 
included all of several -essential" ele=^ts. Xf even one el^ent vas sdssxng 
the re^nse was given a -fail- score. «e recc«end that the user adagt 

this test by both redndu.i the nunber of qaesticos and by relaxing the pass- 
fail criteria. 

taction scores are based on a O-S scale for ±t^. ^ 

a possible range df 0 to 90 for the totals. E^sed in ter.s of j>erce:^ 
of the possible score of 90 tin parentheses), the fractl«, scores fcr students 

^^i^r^ 7fti of the possible i&ile l3ie gwre eiipec- 
with no RfiD work experience averaged 78* or xae jw^.^.-j^' 

_^ «.teiv ten points higher- 33je scope scores 

ienced sssbjects averaged a55>roxiaateiy ten points 

also i^^cate the« are =ar.ed ^f^rences bet-ee» tbe xc e:^ence ^ 
a.e e:,^iencaa sro^. I- this case th. r-test i=aicate= t3>e d^- 
ference is hi^y significant. 

perhaps because the Job Kno»leage Q«stions are fairXy specific, there 
are fe. significant «rxelations between this test and nu*er of co,n:ses. Ml 
three totals correlate sip^ficantly (.27 to .30, .it^. n-,er of courses *n 
Instr«i=tio„al Technology, ^e oaly other signifi^t co^relatix^ns are .243 *>r 
pass-fail ana nurber of Creative Writing course ^ ^"r sco^ and nu*er 
of courses in Philosophy and Hujaahities. 
» 

^ su^arize, the Job fiuastions test does discri-inate iet«e. 

those »ith no H» vor* e^ence and those with so^ e^serience. «e 
^ees are not large, but they are ^tisticalXy signi*»*. '^'^ 
is no evidence that those with s^stantiaiay «re experience perfor. be«er 
on this test, in fact, th^ scores ,f the high e^ence grc-are sUgl^y 

^^wvm> nii^ thr«e itotil scores iave 
/lo.er fhan tho,e of the «.dest e:^ence gro^ *c 

. substantial intercorrelations (.83 between P and P-P, '''^'^^'^^ 
ana .71 between r-P and SO,. Intercorrelatlon of scores '^- ^^^^ 
indicates that the ten ite»s are not highly correlated, onde*. ody 16% of 

indicates tl>a .„„„„„M^ «re significBtt «ri none «»:eea.a 

the between questxon t»uwx.e _ ^ t-t^ 

,-. olsisif strong ^fferences Joetweer. the 

' 50. individ-jal'itaas generally did not sl^ strongs 

• , . t,^ *ofcal scores are dll significant. 

e:g>erienced and inexpeziencei grocpS, but the total scores 
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Given the results, ve would caution the csex: against generalizing cm the basis 
of a student's response to ©nly one or two questions of the type us^ in t2ais 
test. Bo»— er, responses to several questions apparently yield scores idsicb 
have creaihle validity. 

Because of the relative specificity of individual iteas, the instructor 
should exercise care in selecting or writing iteas that broadly sarple knowledge 
in the co2$>etence areas of interest, ^e particular iters used in this test 
were deliberately written to roughly parallel the contest of the siirolation 
test, hence they encoiroass a decidedly narrower range of knowledge than the 
oosoetencies sa35>ied in the self-rating instrument, although the itens sd^t 
be satisfactory for use in testing exit corpetence in the m>ss Developaent 
series or in a coE5>arable course in instructional product developsent, they 
are far too narrow in scope to cover the entire I»SE Program as suggested in 
Figures 1 and 5. 



4. Results for the Exercise in Educational Developaent 

Like the Job Knowledge Test, each of the four tasks of this siiailation 
were scored for pass-fail (P-F), fraction (F) and scope (SC). Each task was 
also scored in terns of the extent to which the simulation directions (DIR) 
were followed. In addition, the four tasks were scored on a total of eleven 
specific eleinents (e.g., the quality of writing of the guideline produced in 
Task 1; the quality of the tryout questions listed in Task 2, the suggested 
revisions in Task 3, the handling of the vislzal eleiuents in the script outline 
in Task 4) . 

The sixteen comon scores (P-F, F, SC and DIR) and the eleven specific 
scores were correlated and factor analyzed. -Siese results are reported in 
Appendix B. Sunsaarized briefly, there were s»derate to substantial inter- 
. correlations aiaong the several scores based on the saae task, but smaller _ 
■ correlatibns a=ong scores for different tasks. However, in contrast to the 
smll proportion of significant intercorrelations found in th^ knowledge 
test, fully one half of the between task correlations were significant. A 
factor analysis indicated that the first principal axis accounted for 34| of 
the trace; in other words approxinately one third of the covariance aiaong the 
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scores vas attributable to a general performance factor. All of the 27 scores 
displayed significant loading on the general perforsiaDce factor bat these 
loadings ranged from .33 to .31. Hence the several scores exhibit substantially 
different degrees «f association with this general perforrsance factor. 

te,en the six fectors were rotated to a variaax solution, we fonnd that 
^ 1 (preparation of product gtiidelines) was associated prSaarily with one ' 
factor aLd Task 4 tontlining a script) was primarily associated with another 
factor. However the scores in Task 2 (preparing for the tryont) and Tas3c 3 
(recosmending revisions based on tryont results) loaded on several different 
factors. «e coJiclude that the sisnlation represents a cooplex of several 
different perforxsance diaaensioxis. 

Hhen we exardned individual iten scores for differences aaong the three 
e35>erience giot^s, we found that the fraction and the scope scores, as well as 
several of the specific scores ^ Tasks 1, 2 and 3 had F-ratios that were sig- 
nificant; however, none of the scores asccciated with Task 4 reached the required 
""significance levels- 

Five total scores were available. In addition to pass-fail, fraction, 
scope, and following' directions, we obtained a consistency total score which 
is the su:a of four specific task scores; one fro, each task^which best reflects 
the test subject's ability to cope with and respond consistently with the infor- 
mation (not the directions) provided in th4 si=talation. Table 7 reports tl^ 
results for these five total scores- . 

AS in the case of the Knowledge Tes^ "there is a prc5>lea with overly 
stringent pass-fail criteria; neither of the e5?>erienced groT?>s has attained / 
even 50% of the possible score. Nevertheless, both of the «?«riencsd group, 
perfona significantly better than the no e^rience grox^. «ie fraction 30ore^ 
is more informative. It indicates that 72% of the possible fraction total 
score was att^ained by the no e^rience gro;^, while the =od«rt:-«¥erie»ce group • 
attained 84% a^d the high e:g«rience grot^. 85%. Scope perfor.«ic6 .averaged 
lower (52%, 70% and 70% respectively), but the difference between , the no e^rxence 
and the^two e,q,erienced groups is larger and highly significant (P<.001). All 
three ofShe grot^s tended to perform well on following, directions, but even 
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Keans for ^ree SSS Sxperience Groops, F-Ratios 
arid P-Levels for ^fests of Differences Anong 
the Groi^ on Siisulation Test Scores 



Pass Fail 

<P-F as % possible) 



fraction ' - 
'IF as V possible) 

Scope 

(SC as % possible) 

Following- Directions 
CDIR-es % possible) 

Consrs'tency with 
Susjulation Infor* 
s3ation*(% possible) 



R50 EXPERISSCE IgVSL 



ISO 



Ito&est^ High 



F-Hatio ' P-Ojevel 



4.38 
(10%) 

25.77 
(72%) 

10.38 
(52%) 

32-46 
(90%) 

23.92 

(66%) 



.5.71 
(43%) 



(33%) 



5.38 



.007 



30.10 ^ 30.56 
(84%) — -- < 8 5 %) 

« — 4 

13.95 14-00 * 

r70%) (70%) 

34.33 
(95%) 

29.67 

(82%) 



5-90 j -O05 



9.17 




.001 



.050 



.002 



/ 
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here tfc^ is a significant difference (P^.GS). FinaUy, both of tie experienced 
^nps were better able to cQoe with tie itsss of sixmlatioi. in&nnatioa than 
the no experience groig). 

5. Llatiocship Between Self- Ratinos, TOxxAedge aad gerfozaa^ce 

T^le 37 of A^dix B rerorts th^ i^itefcorrelaticas asoag fee scale totals 
for the three types of instrm^ents. ' ^ese findings ^/te briefly sa«a=izea as 
foUows. Knowledge as i^easured by all tiilee scores of the Job K«^ledge Qoesticns 
'is modestly,- but significantly, related to all five of the si=alaticn test total 
scores, vith correlations ranging fron .25 to .46. Given the fact that the 
jc^ ruK^ledge Q^^tic^ vere written to cov^r g^r^ cont^t, xoa^y paralleling 
tSr content of the siiailation tasks, the fact that every one of the fifteen 
correlations between th^e two i^trnaents is significant is not surprising. 
Since the Job Kna-ledge Qaestdcns tend to cover relatively general types of 1^ 
ledge and arc dearly not concerned «ith testing for strictly -enabling" Icno-ledge 
that -would be soecifically needed to cor^^lete the si=ulaUon tasks satisfactorxly, 
perhaos the modest level of c^erved correlations (.25 to .*6; average .38) should 
be exoected. lhe§S data do indicate that, unless kno.l«dge content is clearly 
-enabling,- one should not expect to find hi^ corr^ations between knowledge ^ 
^rnance on sirnOation tasxs. ^e average correlation of ,.38 indicates that 
there is less than 15% ^f the total variance of a performance score vhxch as . 
predictable (or exolained) fr6n a knowledge test score. '1^ is hardly a strong 
relation despite the fact that it is statistically signi^cant fro2 ^ero 
cop:elation. 

one 6r rore of the la>owledga test total scores is also significantly 
related to ead, of the three self-rating scales which »ost closely corresponds, 
to the -Job Knowledge Questions content, naBely the Product Develoj-ent, the 
Evaluation, and the Technical Writing self-rating scales. Hoirever, the^e cor- 
relations are also of.quite aodest size, ^e t^o largest one. «»..38 for taxM- 
ledge fraction score with the Product Develoj»ent self-rating scale? ^ .41 
for the knowledge scope score with the sme self-rating scale. »ne of the 
co^elations between the three knowledge test total sco're. and fee xe«ining ferae 
self-rating scales (Kes'eardi, Disse«L-^tic«, and fUn^e^ti are significant. 

-74 . ' 



Sence the Knwl-d^e test does display convergent and discrisainant validity 
in terns of its correlational pattern ^th the self-ratiisg scales. 

Although the significant relationships are extrensely saodest, there is 
wlear ev idence that relevant , jgeneral 3a3c«ledge as saeasmred on responses to 
short answer essay questions is related both t5> self -ratings and to perforrance 
on sinulation test. 

However, there are no significant correlations between the self-rating 
scales and the simulation test total scores- Siis result was not anticipated, 
sinpe there irere significant correlations between these two types of instrunent 
^in field tests of earlier versions of the battery (see Appendix B, pp. 3r2 and 
B-7) - Ihe reasons for this result are discussed in soae detail in Appendix ff, 
po. B-32 to B-34- Stated briefly, the correlations with biographic data {numbers 
of courses taken, products and piibl^tions authored) suggest that the tiiree 
CSB instruneits represent increasing degrees of coqpetence specificity with 
the self-ratings the aost general, the 3cnowledge test interaediate. and the' 
siiMlation test the rost specific. Our guess is that the iters of the JffiDSE 
self-rating scale nay be too gener&l in their scope relative to the partioOar 
perfocaances which are assessed in the sinulation test, and hence, there is 
•no strong relationship despite the superficial sinilarity of content in the two 
types of instruaents. 

c. <x>Kcmsions 

Possibly the nost important practical points to be laade are these: 

All three instruments are valid, both in terms of content validity and 
in terms of their deaonstrated ability to discriiainate between subjects with . 
and without actual product developaent and evaluation experience. 

Even where there are significant correlations between types of instru- 
nents, the relationships are very nodest; hence each instruzaent tends to provide 
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_ different kind of vali*^ infomation about coEoetencfe- fince n^^rofessicnal 
training prbgrars seen to rely heavUy on paper and pencil tes^ of knowledge, 
it nay be comforting to kn<- that significant correlations were found between 
t2,e essay test scores and both the self-ratings and tha siiailations. However, 
the'correlations are so «nall that one needs to question the utility of an 
assessr>ent systen that relies exclusively or alrost exclusively on reasureaent 
of ^knowledge- * 

Because the several types of assessrenfc instnnents are reasonably 
independent of one another, and yet all are valid, their coribined use should 
increase the overall vaiic^ltv of coi^^tence assessr^t decisions. (See i^- 
di^ ^, footnote on p- B-35 for one exanple.) Moreover, inclusion of ot^er 
types of inforzation (see 12 and 14) is recormended. 

Conoetence assessment is obviously a "sarpling- proposition. Our field 
test results suggest that there are problems which deserve special attention. 
Given the generally low correlations arong iteas within knowledge test and 
witMrt- the sinmlation t^t, one nust exercise cauUon in caking generalizations 
except where the nu:dt>er and scope of test items ^re sufficient to adequately 
represent the competence area being assessed. The high degree of specific!^ 
of ^ task in the simulation test, and its factorial coniplexity are indeed 
troubling. Our concern is that assessment based on performnce on one or 
only a few ^imilation tasks i=ay lack substantial general validity. Apparently 
a similation task is analoaous to a single test item in terms of reliability 
arfd validity; we need a sarn^le of several tasks to achieve useful reliability 
and validity.** The four tasks coi:?>rising the aarrent sioulation are^liaosi^ 



' . nt is quite conceiva ble that different results xaight be obtained 

Simulation tasks or highly specific rating items wxth essentially the same - 
competence content. 

**Nofce however that several different elements'may be scored on each task, 
sS:^StiS^n.istina of several tasks may yield severax different 

total scores. 

64 

ERIC 76 



too nuch to handle at one test session, and yet this aaount of simulation per- 
&mancc- inforration seems hardly sufficient. Several Eore tasks are probably 
reguii^a to provide even aininally satisfying sas^lings of even the educcitional 
product development cor5>etence dbaain. / Given the substantial costs in student 
and instructor tii« that need to be invested in this kind of perforrsance assess- 
nent. we doubt that simulations will be attractive unless they are built int o 
the instructional progran and treated as signific ant learning opportunities 
(with adequate instructor feedback and counseling) rather than as solely assess- 
oent Exercises. Please note that we believe the simulation tasks should be 
a port of a cor^^tence based program. Many of the students who took the 
simulation test indicated that it was bne of th^ few times that they were actually. 
challenged to see what they could "do- rather than just what they "knew" 
about educational product development. And, despite its artificiality in 
terms of tine limits, lack of team interaction, inability to ask questions or 
seek assistance, rost RSD professionals considered the simulation to be a 

valid and zneaningful challenge- 

•J 

Finally, we must realize that what is presented here is a deliberately 
i^pen-~aise^nent system, rne assessment model and the resources illustrate 
what can be done. For those who wish to establish a DDSE training program 
based on the Far West or similar training materials, the instruments and guides 
contained in the following appendices will provide a good foundation. But 
there is a clear challenge to make significant- additions or modifications 
to the resources we have been able to provide. 
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The Far West Consortiim for DDiE Training has created a set of instruc- ^ 
tional nodules *ihich prov4<Je: a ^re of curriculum resources for a ccrcpetence- 
Sased prograa for the training of entry-level professionals in educational 
"development, dissemination, and evaluation (DDiE). As part of an assessment 
systero, a DE&E Diagnostic Battery was developed' to help students znd instructors 
to detennine the students' levels of competency in order to plan prograns of 
InstructiqnjtaiJored to individual needis. Because self-ratings have proven to 
be a viable and inexpensive nieans of obtaining this c'cffpetehce Inforaation, 
a -self-rating instrument, the DDiE Diagnostic Iiwtrunent, is the i»ost Important 
part of the Diagnostic Battery. .Paper and pencil tests of feiwledge and job 
sample tests constitute the other elements of .the battery whia my be employed 
as validity or calibration checks on. the self-assesssc^t instnuient.. 

This report describes the field test, data analysis, and revision of the -. 
DDiE Diagnostic Instrtfnent, The instrument consfsts; of a swpleor72 corape- _ ' 
tence statements which -have been drawn to provide equal representation of the 
.three major work contexts : development, dissemination, and evalu.ation (24 items 
each), and simultaneously to provide equal representation of six process skillsr 
collecting infomation, analyzing, planning and designing, |>roducing, evaluating-, 
and cpmnunicating (12 Itsss esch, four in each of the three contexts). The- 
studfent is| asked to rate each iton on a seven-point scale that a)mbines knowledge 
and experience to provide a behavioral reference for raa'king.dudgments. about the 
attained level of proficiency. . \ ' ■ 

,The fieTd test of the Instrument was- conducted at several academic institu- 
tions 'and RiD agencies throughout the country. A total of 78 subjects provided 
usable data (62 under-graduate and gif-aduate students and 16 developers and senior 
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developers in RiO agencies). Analysis of the data indicates that the DP&E 
Diagnostic "instmnent jteas are highly intercorrelated but that at least three % 
or possibly four subscales can be differentiated. Cluster analysis resulted 
in four well -def^ijied .clusters, three associated vrith the.DD&E contexts and a 
fourth derived primarily frcm items appearing in the DiO contexts relating to ■ 
publication, production; and public relations. However, correlatiqt^ and factor 
analysis indicate that this last cKster is highly correlated with and exhibits 
a factor pattern similar to the disseciination cluster. The factor analysis re- 
sults suggest that only three factor scales are needed: 1) developtaent, 2) field 
test and evaluation, and 3) production, dissemination, and marketing. 

Neither the clus*-r analysis nor the factor 'analysis resulted in a group of 
canpetence'statenentS associated primarily with any one of the process skiiisy- 
howevar, both types of analysis, indicate that the newly defined DD&E factors 
exhibit different patterns of skills which appear "to be meaningful. The evalua- 
tion factor is most prominantly associated with: collecting information analyz- 
ing, and evaluating; the production, dissenination and marketing factor with: 
planning and designing, producing, and cotmiuni eating; the development factor with: 
analyzing, planning and designing, producing^ and evaluating. 

The analysis indicated that the items which had been placed a priori in the 
development context , were in fact a mixture of all three DDiE competencies, 
consequently, the number of items associated purely with development is closer - 
to nine in nunber rather than the intended 24. 

Given the predominately student population^ of the field test sample (62 
of 78 subjects), the average competence level and the distribution of item means 
seans acceptable, since the average proficiency level for the'entire instrunent 
is at the midpoint (4.0) on the seven-point scale, and the great majority of the 
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iten prof icien"cy .levels are between 3.0 and 5.0 (61 of the 72 itens). Analysts 

of variance between the student ar.d the developer groups indicates that the 

majority of the items and all of the kcales discriminate between these groups. 

in tpnns of the proficiency scale, the average difference is ^..27 proficiency 

levels. The means for the majori^ of the competency statements for the student 

group were below 4.0 which corresponds to the statenent: , 

"I have a' general understanding of this activity and have had some 
experience with it, enough so Tcan do it if \ have either detailed 
instructions or close supervision." 

On the other hand, the majority of the competence statenem: means for developers ^ 
were above 5.0, wHlch corresponds to the statement-: ^ 

"I have enough experience in" perfonnihg this activity to do it if 
given enough general supervision or general instructions. 

This clear, meaningful, and highly significant difference (p<.0001) provides 
further evidence that the instrunent is able to discriminate between groups with 
different amounts of relevant experience. (Previous pilot studies have shown that 
self ratings made by educational developers correlated with supervisors ratlngs'of 
the same sets of competencies, .73 for one' scale and .88 for another, thus Inaicat-^ 
ing that experienced developers and their supervisors exhibit substantial agree- 
ment regarding the average levels of proficiency.) 

In terms of levels of self-rated proficiency, both the students and. the 
developers reported relatively higher levels of proficiency in ^development, 
tpllowed closely by field test and evaluation, and relatively lower levels of ^ 
proficiency in publication, production, and public relations comi^tencles. Both 
groups rated caimuni cation skills highest and planning and designing skills lowest- 
The difference between the two groups was at least, one proficiency level for all 
of the subscales (contexts,'skills. cluster analysis scales)-, with the greatest 
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differeilces appearing on the developaent context; scale, the field test and 
evaluation cluster scale a^i on the analyzing, thf collecting information, 
and the evaluating skflls scales. The saallest differences between the t»rt) 
groyps «ere on the disseainatton and narketing context scale, "Uie ^related) 
publication, ^^njduction, and oublic relations cluster scale, and the coencainica- 
tion skills scale. In the case of coonainication, the relatively 'iESjal 1 difference 
in proficiency level seaas to be attributable prioarily to a (relatively) high 
level of proficiency in the student group irith respect to oral and written 
comnunlcation. in the tSse of the disseaiination anS marketing context scale 
and the publication, production,, and marketing cluster scale the smaller dif- 
ference between groups is primarily attributable to the relatively lower levels 
of pro/i"ciency reported by the developers. 

Given the results of thf data analysis, it was decided to retain the con- 
tent structure of the instruaent (matrix of three contexts by six process skills) - 
but to revise a niinber of itecs so' that the three ODiE factors would be more 
clearly and evenly represented and more'" closely associated with the three context 
cbTumns^of the response form. In addition, an eighth level was added to the 
proficiency rating scale to separate the expjert from the highly experienced- 
Appendices to this report include copies of both the original and the revised 

4nstruiP3nt. . ' '^'^ 

Field testing of this revised instrument will be accomplished In connection 
with the field test of the 0D5E AssessmenC Battery. This will perait correlation 
of self -reporting infonnation with Jcnowledge test scores and scores on a series 
of ODSE job ssniples. , , . . 
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The DD&E Diagnostic Instruaent is a guidance device to be used in planning 
a student's prograp of study in conjunction with the use of the DDS£ cospetence 
based learning resources.^ The content of the instniaent Is based on the DDSE 
Coapetence Matrix (Figure 1), which structures the Isportsnt activities in develop- 
nent, dlssjsmi nation, and evaluation (DDiE) in tenas of three contexts (0,D&E) 
and six process skills, "nie Diagnostic Instruaent "is caoposed of a sample of 
four items (activity statements) for each cell of the aatrlx. The task' for the 
student (pr other rater) is to rate the 72 Items according to a seven-point 
Proficiency Scale. The Proficiency Scale cofcbines level's of knowledge, and 

- ^perience. ' ■ ' ^ . • ^ 

LEVEL- OF PROFICKHCY 

ll I have no specific knowledge about this adtivity nor experience vith it. 

* 

2. I have read about or seen this actlvi^ performed, but have no experience 
WH^^^t and* don't really understand it, 

3. I have studied this activity jor have frequently- seen it perfoimed and liave 
a good understanding of It,, but I have not yet done It. 

4. I have a general understanding of this activity and have some experience 
with it, enough so that I can do it If I have either detailed instructions . 

; or close supervision. 

5. I have enough expenence in performing this activity to do It If given 
enough general supervision or general Instructions. 

6. I have enouah knowledge and experience with this activity so that I can 
perfona this task quite satisfactorily without supervision or job aids. ' 

^ * • 

7. I have had extenslvs experience with this activity, and can perfom It 
quickly, efficiently, and do a top quality job. 



^Readers who are not familiar with the rationale and technical awro|.ch 
for developoent of this InstruKsnt are referred to: f*"^^"- 
ZawII ng^? Assessment Instnynts for Cfntp ^tgnce-Sased E<fac«t1«i^ 
SI pSK? ;?Ln.> D^veloanent. UU^usion and Evalua tion T rai ning nro a g^ 
(Stn francisco, Ca. ; Ur Vest U boratory lor fcducationat Kesearck ana 
Oevelopweht, Movember, 1973), 

• ". S4 ■ 
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Kpte: RoBian numerals tally itesns by Cluster Analysis (see page 5): 

Cluster I - PubJicatfon, Production and Public Relations; Activities. 
Cluster II - Field Testing and Evaluation Activities. 
Cluster III - Planning and Evaluating Djsseminauon Actixttxes — 
Cluster ly - Development Accivities 



The instninent is available in tifo fonss: a conventional rating fcna and 
a Q-S(rt deck. In the Q-sort fora each of the seven proficienQr levels has been 
separately listed on a 5 x 7 Category Card and each of the 72 activity state- 
cents appears on a 4 x 5 Itaa Card- The users are instructed to place tte 
Category Cards on a table in front of then and then sort the deck of 72 Iten 
Cards by determining their level of knowledge and experienra for the iteca, and 
then placing the Ites Card on the appropriate Category Card. There is no time 
limit. * • 

A record forn, organized in the same form as Figui^ 1, but with labels* 
brief statements of the content of each iten, and space for recording the scores 
is provided. Khen canpleted, the Diagnostic Instrgsent record form consists of 
the rating for each item, along with cell, coluinn, and rcw sioinaries, irfiidi 
provide a systessatic representation of the student's knowledge and experience 
in DD&E- The completed instrument is useful to students and advisers for 
identifying a learner's strengths And weaknesses in DD4E Conpetencies. It 
also provides a -structure by which student and adviser can discuss DDiE Cocnpe- 
tencies in relation to the student's, particular learning objectives or interests. 

The following report describes the field testing and 'subsequent revisions 
of the DD&E Diagnostic Instrussent. . ' _ _ 
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The DDSE Diagnostic Instncsnt is self-adainlstered and takes approxi- 
cately one hour to canplete and record. A list of the itsss in the Diagnostic 
Instnxaent and the rating scale can be found in Appendix A. A total of 73 
subjects, representing a wide range cf ^xpeHence and training in educational 
developHfint, completed the instniaent. We have grouped the test subjects 
into four categories according to «ieir educational background and training. 
A description of the four categories and the nicber of subjects In each 
category is shc*fn below. 

itebef ~ Description of Category 

11 Students recruited frcsp a local university «ftose back- 
grounds wem siailar to that of students wa expect to 
enter the DD&E program. They **sre seniors or graduate 
students majoring in education or majoring in the , * 
behavioral scien(^ with an interest in education. 

51 Graduate Students (Hastei^ and Ph.D. candidates) enrolled 

in educational research and developcaent courses. These 
subjedts were recruited by their professors at Tenplfe 
University, San Francisco State University, Michigan State 
University, and UCLA. 

12 Developers ^ Masters level perbonnel with at least one 
year of experience in educational ^^evelo^3Blent, or-persons 
with Ph.D.'s In education who have sane association w?th 
educational development or who are presently developing " 
educational products. Seven of this group were jenrolled 
in inservice training in DDiE. 

4 . Sen ion Developers - Hi ghly experienced in developing 

educational prodacts and programs. These personnel are 
presently working in the field. They have published jn 
the field and/or produced products that have been dis- ' 
trifauted and are being used. . - 

~r~ * , ; " . 

78 TOTAL 'J: 

.Please note that this group of, s^ibjects is predosinantly a student popu- 
lation, which is aopropri?te since the Diagnostic Instrument fs intended Tor 
professloiral education settings. The reader M cautioned to keep this fact in 
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Qind while reading the following analysis. The results obtained say not be ^ 
generalizable to larger groups of experienced R&D professionals. 

CATA AHiSLYSIS 

■ 

Cluster Analysis 

A cluster analysis of the data was perforaed by ase-^f the BC T3Y coa- 

putsr package available through the University of California at Berkeley. 

Tnis data analysis technique groups together variables (itaas) on the basis 

of their similarities and differences- It results in clusters of iteas which 

2 

are similar to each other and similarly different froB other iteas. The 
cluster analysis technique was used to deteroine if the underlying structure of 
the Diagnostic Instruuent (the matrix) would be replicated by the ecpirical 

grouping of iteas- 

Four clusters of items resulted. The clusters are nsost similar to the 
columns (functional contexts) of the DDiE Matrix. Cluster I is a grgup of 
12 itesis which describes publication , production , and public relations acti- 
vities- Cluster II is canposed of 25 items which describe field testing and 
evaluation activities- Cluster III, a disseraination and marketing-cluster, 
is a group of 24 items which describe tasks associated with planning and evalua- 
ting dissanination activities. Cluster IV is' composed of 9.1 tens describing 
. ^velopment activities- Only two items did not appear in a cluster- A list 
of the i tens- in each cluster can be found in Appendix B. It Is noted that none 
of tlie clusters clearly associated with a single process skill (rows in 
Figure 1). However, the clusters are associated with different groups of process 
skills. The items in Cluster I involved primarily the process skills of planning., 
production, and communication. Thej'tems in Cluster il involye all six skill 
arCAS, but are most frequently represented by analyzing, eyaluating, and collecting. 
information skills. The Items In Cluster lit also Involve all six skills, but 

d ^Robert C. Tryon and Owlel E- Ri.npv. -Cluster Analysis (Hew Yorfc, N. Y.: 
ERJC HcGraw-^mit Book Co. i 1970>, ppv l-5> 45^8. , 
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are cost represented by corcunicating , evaluating , and collecting infciraticn 
skills. The iteas in Cluster IV are unique in the fact that this is the only 
cluster in whidi all itens are associated with only one context (developcnent) ; 
however, five of the six process skills are represented and in approxinately 
equal proportions. 

Field Test and Eoahtation Coxpstenaies 

The cluster analysis results Indicate that we can differentiate D9SE coa- 
petencies. The largest and most clearly defined cluster is composed of field 
test ano evaluation activities. Nineteen of the 25 items in this Cluster (II) 
are in the <^ield test and evaluation coluan while the remaining six are in the 
development colunn. Four of these six itesns specifically mention evaluation 
or field testing- Conversely, only five of the 24 items a priori associated 
with field testing and evaluation failed to enter Cluster II. On examination, 
two of these five itens (46 and 48) contained no wording that would permit 
association with any D,0 or E context; and these were in fact the two items 
that did not appear in any cluster in the data analysis. 

The first five of the 25 items to enter Cluster II are listed below in the 

order in which they were added to the cluster (see Appendix B for the entire list): 

42. -Evaluate test instruments using data collected in tryout and-revision 
cycles in order to recommend instrument revisions for the final field 
test. 

37. Hake revisions in test instruments based on evaluation data. 
17. Translate field test data into reccnmendations for action.' 
39. Prepare a test administration manual . 



7 Provided with field test data on instructional materials, examine low 
gli!! scores and detennine if they indicate problens in test construe- 
tion or instructional materials. 
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Dissesinaticn end Mca^tir^ 

The a priori set of 24 itsns placed in this coliEJn appeared in two clusters 
(I and III), but 18 of the 24 iteas were associated with Cliister III: Planning . , 
and Evaluating Qissgnination Activities. Cluster III also Involved four itens 
«hich had been placed in the Developaent colusn and two In the Field Test and 
Evaluation ralunn. It is notable tiiat four pf these six "outside the coluan- 
iteas have "corownication" verbs (confer, discuss, interact, and coanainicate) 
which suggests that conraunication, especially. informal, oral, two-way conraunica- 
tion as well as planning and evaluating dissesainalion activities are inportant compo- 
nents of this cluster. The first five itens to enter Cluster III in the analysis were 

57. Outline factors which inust be considered in dissesninating jn^o^a- 
tion about an ec&Jcational product designed for a specific target 
group. 

66. Carry out an evaluation of the effectiveness of a daaonstration of 
and educational product. . , 

59. Plan interviews with potential users for the purpose of determining 
a market for your product. 

- 55 Review alternatives for the design of a product in tenns of possible 
problems in installing or maintaining the product. 

65. Determine the thoroughness of distribution which occurred in dis- 
seni nation of an ^ducational product. . * - 

. -Development ^ 

The items which had been placed a priori / in the development coTumn turned 
out to be a highly mixed, lot, with only nine of the- 24 actually entering Cluster . 
IV: Development. Four itans with "communicatida" verbs ^M. 22 , 23„and 24) • 

• were associated with Cluster III: Dissemination and Marketing. Six more itens 
were associated with Cluster -II: Rpid Test and Evaluation, and several of these 
items employed "evaluation" or "field te^t" references. Finally, five items were- 
associated with-Cluster I: Publication, Pro duction and' Public Relations. Perhaps 
this mixed resulto's only a reflection of our Inability to write and place 



8 , 

"developraenf-iteas effectively; however, it may reflect sooething important 
abffut the variety of different skills that are actually associated with 
developcsent. , 

The entire list of the nine develoosent iteos, listed in the order they 
were added to Cluster IV are as follows: 

11. Arrange learning activities in a sequence to facilitate learning 
or mastery of objectives. 

15. Write exercises which the learner should do In order to ssaster 
concepts or principles In an Instructional unit. 



19. Provided with product specifications, review a product ahf 
docunehtation on product development and field testing to 
determine If the specifications have been met. 

8. Categorize instructional objectives In terms of a taxonomy 
(e.g. , Bloom's Taxonomy of the cognitive domain or Gagne s 
conditions of learning). 

10. In producing specifications for Instructional materials deter- ^ 
mine an appropriate format for the -materials. — 

16. Write copy for instructional materials -from product specifica- 
tions. ' £^ 

'-' 

2. Locate existing methods or strategies which can be used in 
potential product development- 

5 Given a problem statement, information o^ the history of the 
problem, objectives and possible solutions, write a 20-page 
proposal for solving the problem, including a rationale for 
the approach and a development schedule. 

3. Prior to conducting a survey of the literature, prepare a . / 
search strategy. 

Publ-ication^ Produqtion and FvibVic Eelations 

•This set of competencies was unanticipated by our original competence ^ 
matrix, but emerged as a well defined' and quite meaningful set of twelve itemsi 
consisting of six items, from the Dissanination and Marketing column, five 
Items from the Development column and only one Item from the Field Test and 
Evaluation column. The first five items in this cluster ^are listed below . 
^n p\€ order they were added to the cluster analysis: 



60. Prepare* specifications fpr audio^visual mater-EalS which will 
used in the disseKiination effoi*t. 

61. 'Write public relations scripts for film and slide shows. 

62. Write Dress releases ±o disseminate information aboilt a 

new product. . ' * 

56. H?kc arranganents to secure copyrights and copyright clear- 
ances* where needed. 

13_.__lijcejsjtep3_i»-^s5ureJ;hat_sach_$kto5li^ _ 
copyright r§leases or patent protections are secured when 
appropriate. " ^ . 



Levels- and ^istH H»»t^«"^^>f j>rofic4encv Ratings^ — -~ -— — ;—- 

The average competency rating for the Js'subjects over the 72 items was 4.0" 

^the itedian was also 4.0) , which oirresponds to the rating: 

"I have a general understanding of this activity and 'have had sqne 
experience with it, enough so that I can do it if I have either 
detailed instructions or close supeiyision." 

Sixty-one of the items wel-e within one proficienor level of this average (3.0 to 

5.0). Only five itens had average ratings over 5.0 and only six had ratings ■ 

under 3.0. The average rating and the alpha coefficient of reliability (internal 

consistency) is shewn for the four cluster* analysis scales below. 

TABLE 1 . ' . . 

, " ' CLUSTER ANALYSIS SCP^ES 



Cluster ' 

I. Publication, Production, and Public Relationis 

# 

II. Field Test and Evaluation . 
in.' Disseniinatioa (Planning and Evaluation) o '* 

4 - 

!y. ■ Pcyelopment 



Items 


Average ' 
Rating 


Alpha 


12 


3.6 


.91- 

« 


- 25 


4.T 


/96 


24 


3.9 ' 


*95 


'9 


4.3 


.90 • 
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All four of these scales exhibit high internal consistency and satisfactory 
levels which are neither too high nor too low in terms of average proficiency 
levels. Exsainatipn of the frequency distribution of proficiency le-zels for 
individual itens did indicate that on nine of the items, over 20 percent of , 
. the subjects fated theniselves at .the 7.0 (highest^ level. Interviews wfth 
several of the highly experienced subjects, indicated that they believed there 
was a need for an ejghth/expert" level of proficiency. An eighth level has 
been added to the revised scale (See Appendix A-Ot 

DiscrriMnaUon Among &oup8 With Different A:nsu/2t8 of Experience 

< * 
-As part of the iteni analysis, each of the 72 items was treated as a depen- 
dent variable in a one-way analysis of variance. For this analysis the two 
groups of students were canhined (H=62) and the two groups ^f developers were 
combin^ed (N=16). Means 'or the student group and the developer grou)), F-ratios 
(=t^) and p-levels for each ifen are reported in Table 2. All 7Z'*items exhibit 
higher means- for developers than for students, and for 51 items the significance 
levels of the differences are .05 or less." 

The development context items and the field test and evaluation context items' 
each average approximately 1.4 points higher for develflprs than for students, 
with students averaging below 4.0 and .developers averaging above 5.0. Only two 
development items and four field test -and-evaluation 'items did not display 
significant differences (p<. 05). ^y contrast, the disseihination and marketing 
context items exhibited markedly lower averages for both groups, and a. smaller' ^ 
difference (1,03) between grouos. Only nine of the 24 items disgl^yed, significant 
, differences (p<.05). After reviewing the context -of the dissemination and market- 
ing itens, it is our conclusio;i that developers general!/ aren't as^rofic1«nf- 
in this area and, therefore, are not as fariibove student^ in proficiency as they, 
are jn^other areas. ^, % 
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ABtREVMTED ITBt Ca<7£K7 



MEWS 
SnjOEHTS 



1. ObUfn infoneation on^roblaa 

2. Locate strategies for dcYelopnent 

3. frepire'* seirdJ s^»tegyr 
fwluateTrffoSatTroTa- reJevMce 



5. Ifrite a 2p-p»9e prt^al 

6- Use instructional theories in design 

7. Reifiew field test data"f or revi^oo- 

8. Classic instructional objectives 



9. Plan '% budget 

lOi Specify fomat of naterials^ — 

11. SequMCfr learning activities 

12. Estiaate needed production cater^als 



13. Secure waivers* releases, etc^ 

14. Confer witt specialists when needed 

15. ' Write instructional exercises. 

16. Write copy fro!LsP?9tt!«*yj*l* 



17. Jfake recoi«Midations fron field data 

IS. Conduct a case of a progriB 

19. Check product against specifications 

20. fnfonKlly tiy but a product 



21. ^ Write a position paper, 

22. Discuss a product with user groups 

. 23 . Intel-act in a staff ■ceting 
J^CdBwnicate.proAict -specifications 



2.97 

4.42 

t735- 



4.06- 

3;44 
4.0T 



KEANS 
CEVaOPERSJ 



5.62 
5.44 

-ft;44- 



3.50 
3.« 
4.37 
V.il 



an 

4.S3 
3.76 



3.35 
3.95 
3.76 
4.i§ 



\ 70TM. - DEVELOPitftiT 



3JSZ 
AM 

s:n 

3.58 



5.00 
5.75 
4.94 
J.50 



4.S6 
5.31 

5.61 



-37?4- 
5.19 
5:56 



4.75 



5.50 
* 5.00 
5.38 
6.19 



3.» 



5,00 
5:62 



5.39 f 



F- 

WTIO 



5.43 
^1.88 
8.50 



^.04. 
16.75 

6.59 



4.02 
9.40 
2.70 
11.59 



4.90 

3;96 



3.04 



13.37 
4:45 
10.31 
12.03 



4.27 

'9.9^ 

8y*i 

^^*^^4 



20.12 



.003 

.'goi 

.005 



.026 
.000* 
.004 
.012 



.046 „ 
.003 
.101 KS 
.001 



fe69:— .01T 



.028 
.048^ 
.081 KSI 



.001 

i036- 

.002 

.001 



.040^ 

.Obi 

7 

*obo 



.0001 



^11 ValOes ttbted .000 have PtUyels of 
JjS '- llot signfficifit tt 51 leveT 



,0005 orltss * 



94 • 



M ^ 



0 * • 
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TABLE 2 (confd.) 




MEMS 

STUDENTS 



Prepire a coding scheac 
Organize sVitisticilMU 
Interpret scatter plots 
Insure privacy in data collection 

-t)1scussnntej:nal-i external yalHity 
FomiTate evaTuation questions 

r Analyze test outcowcs 
Decide If tests fit evaluation pTan 

Oetenoine validity of your test 
•Design a project isonitoring systea 
Tlan controi of extraneous variables 
Set cHteria for field tesr sites 



37. 



Use evaluation data to revise tes ts 
Adjust test procedinres «hen needed 
Prepare a test administration nanual 
Discuss standardized interviews 



Decide If 'Statistics are suitable ^ 
Evaluate test instruncnts ^ 
.Find intenial goasistency of a test 
Identify ^n evaluation's purpose 
pfepw an article for-publication 
Give a short speedi or oral report 
Prepare simple evaluation reports 
Prepare graphs to display data 
TOTAL - EVALU ATION 



3.86 



RATIO lUYEL 



16.16 



,068 NS 

1:269 NS 

.ooa 

.096 HS 



' KS 



• All Values tabled .000 have P-Leyels of .0005 or less. 
Hot sighiffcant at 51 level , . \ ^ 



T«LE2 {crtt'd.) 



ASBREVIATEO ITEM CttdOfT 



OISSEMUIATIKl t lW«£nSG 



fCWS 

STUOEKTS 



{«-«) 



KEMS 
DEVELOPERS 



F- 



49. Collect dau on fnstalUtion ceste 

50. Design Mrieting study ^att forms 

51. Use. infooMtion services 
^27-Xei»stnjct an annotated bibliosr^y 



53. 
54. 

-55. 
56 

57. 
58. 
59 



Identify character of target grwp 
Evaluate «rket research tediniques 
Detemine problMS In instalUtipn 
Secure-copy right clearances. 

List iaportant disseainatioft factors] 
Design a public relations activi^ 

Oetenaijie a naricet for your proAict 

60. Prepkrfe' specifications for A/V aaterial 



61. KrlU scripts for fil» * slide shows 

62. Wrltlliress releases aboit a product 

63. Prep^ a seaple product 

64. Write spscifications for user canual 



65. Evaluate the product's distritullon 

66. Evaluate effect of a denonstration 
87. Identify parts of a fcariceting study 

68. Interview users abo ut product use 

69. Establish cooperation with users. 

70. Deliver ^ oral presentati«» 

71. Dewnstrate the use of a' product . 

72. Translate dat* into vertaf fom 



TntI>L - DISSBilHATIOa 



2*95 
.'2.79 
3.^9 



ti'25 

5.11 
S.12 



1.05 



3.76 
2.M 
3.08. 

3:^3 
im 

3.34 
3.02 



3if) 
5.00 

3.50 
3.75 



1.17 
MM 

1.11 
^:2f 



.m MS 



3.19 
3.16 
^.5t 
3.74 



S.M 
4.«l 



2.77 
3.77 
3.57 

i.m 

4.15 
4.11 
4.53 
3.81 



3i44_ 



3iis 
4(06 

5.31 

4;38 



4^7. 



V.«4 

J0.I5 



3i?7 

5^97 
7,34 



.097 US 
>1M 
.000 

V14&KS 

ii)72:KS 

.14^115 



US 

>003 
.t>70 1ft 



:053 IIS 
.052 KS 
.214KS 
.dl6 

.20ittt 
,»35IUS 



♦ - A13 Y*lu«5 tabled .000 have p-Uvcls of 
ns - Kot significant at 5X level 
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It is ROted that where 20 of the dsveljpsent ftens and 16 of the field test and 
e^aluatitsn iteriS are rated 5.0 cr higher by the developer group, only 8 of the 
disseiinatlon and :T3arketing itens are rated above 5.0. 
jinalysis of Scales 

Original ly the DD££ Diagnostic !nstnrr£nt was designed to provide the three 
aintext sasres, six procsss skill scores, ar<d a total score, but the cluster 
analysis indicated that there were four fsajor clusters. To detenaine the relaticai- 
ship a:5ong the several possible scales, the scale scores were correlated and factor 
analyzed. The group to which the subject belonged, as described on p. 4 , Kas 
included &s a variable roughly indexing developaent experients. Table 3 presents 
the results including scale neans and standard deviations, the average profi- 
ciency rating for itess in the scale (itea mean), the table of inWcorrelations, 
and the results of the factor analysis. 

Because the number of itenis in the scales is different, the iteni means have 
been included in the table to facilitate a)n3parison. The iten aean for the 
Total Score is 4.02. AT!ong the three context scales, the dissenination and market- 
ing scale is markedly lower (3.65) than the other two scales (4.28, 4.14). ftsong 
the skill scales, coasnuni cation skills are highest (4.68) followed by collecting 
information (4.12). This is not surprising since roost college students have some 
proficiency in at least seme of the items included in these two scales. The low- 
est mean is for planm>g skills (3.66). A-nong the four clusters, we find results 
''that not unexpectedly reflect those obtained for the context scales. Again, 
development is highest (4.3), but the low cluster is publication, production, and 
public relations (3.26). 

The standard deviations (S.D.), also reflect the number of items in the 
scale. We hsven't included iten S.D.'s, but it can be noted that the S.D.'s for 
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trtr three conteict scales (eadi 24 Itetr.s) are qjite similar, as are the S.D. 's 
for t»ie six skill scales {eat* 12 iter^s). 

Tre table of intercorrelations indicates that all of the scales are highly 
^ntercorreiated. ^ctcng the three context scales, the lowest correlation is .58 
between the disseninaticn and fnarketing and the field test and evaluation scales. 
Both of these scales a)rreiated with the developn:ent scale in the high .70*s. 
All of ti:e process skills correlate .71 or higher, feong the cluster scales, 
the lowest correlation is .58 between the field test and evaluation cluster 
scale and the publication, production, and public relations cluster scale. The 
.83 correlation between this latter scale and the disseiuination cluster scale 
indicates that these two cluster scales are substantially correlated (and in 
fact appeared as a single factor in the factor analysis). The reader snould 
recall that the three set* <;calos (context, process skills, and clusters) 
are all derived fron the seize set of 72 itens and thus should show substantial 
intercorrelation, but it is perhaps surprising that the lowest between scale- 
type correlation is .61 between the field test context and the publication, 
production, and public relations cluster . 

Hone of t!ie scales correlated highly with the group index (l=student, 

2-graduate student, 3=developer, and 4=senior developer), although it should 

e 

be recalled that 51 of the 78 subjects were in one group, graduate students- 
All of the correlations between scales and the group index are between .30 and 
.45. 

A principal components factor analysis was performed. The first factor 
extracted 79 percent of the trace, and all loading for the scales were above 
.85, indicating that a single general proficiency factor could account for 
fnost of the covariance among the scales. The second factor, accounting for 
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se^en percent of the trace distinguished asicng the three contexts \nth loadings 
of .41 cn field test and evaluaticn, .02 on developcent, and -.41 on evaluaticn. 
The third factor accejnting for 5 percent of the trace was associated alcost 
solely with the group index. It had a loading of .79 on tte group index. The 
only scales with even nodest loading were variable 2 (-.27) and variable 11 
(-.23), both concerned with field test and evaluaticn. A fourth factor, 
accounting for only three percent of the trace, was defined in terns of 
differences between development (variable 10 loading -.48 and variable 1 loading 
-.21) ar*d publication, production, and public relations (variable 13 loading 
+.22). 

These four factors, accounting for 94 percent of the trace, were rotated 
by the Variiaax r,ethod (Kaiser) to obtain the four orthogonal factors reported 
in Table 3 . Factor ! is the developEent factor, i^ich is best identified 
by the .79 loading on variable 10 and the .59 loading on variable 1. Factor II 
is the field test and evaluation factor, identified by the .91 loading on 
variable 2, and the .90 loading on variable 11. Factor III, a production and 
dissemination factor, identified by the .90 loading on variable 3, the .84 
loading on variable 12 and the .87 loading on variable 13. Factor IV is alrnost 
solely associated with the group index variable, which loads on this factor 
.97. The four rotated factors account for the following percentage of the 
trace respectively: 16%, 312, 39%, and B%. Excluding Factor IV, we see that 
developrsent (Factor i) is least well represented among the three DD&E factors. 
This result Is paralleled in terms of the loadings of the Total Score, variable 
14, on the first three factors (.39, .61, .67). These results strongly indicate 
that the DD8E Diagnostic Instrument is deficient in statements that reflect 
cocnpetencies of the type contained in the developtr^nt cluster. On the other hand. 
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both evaluation {Factor II) and dissenlnaticn (Factcr Ili) are Kell represented. 

Turning to Individual scales, we note «iat the de/elopnent ccntext scale 
loads approximately evenly cn a]Xof the first three factors. This result 
is cor.parable to the finding disojss&j cn d 7 regartiing the cluster analysis. 
In the current DB&£ riatrix, the de/elopn^nt colunn is a fixture of several types 
of co:!:petence statei^ents. By ccr":*-ast, variables 2 and 3 exhibit relatively 
pure, tut not axnpletely uncxints^inated factor coxposition. 

Skipping to the four cluster variables, because they are snost closely 
related to the context variables, we note that the factor coniposition is well 
defined, but not quite so pure. The developrnent cluster, variable 10, does 
have a rnuch stronger loading on Factor I (.79) than cn the otfjer two factors 
(.37 and .4i). The field test and evaluation cluster (variable 11) has a 
loading pattern quite similar to variable 2, but this sh(^ld be expected since 
they share a large nuniber of itesns. Reflecting the .83 a)rrelation between 
variables 12 and 13, we find that these two variables exhibit highly similar 
patterns of factor loadings; both heavily loaded on Factor III, and with much 
smaller loadings on Factors i and II. 

Turning finally to the six skills scales, we see that the collecting 
infonnation skills scale, the analyzing skills scale, and the evaluation skills 
scale show highest loadings on Factor II, evaluation; while the renaining 
three skill scales, planning, producing, and conmuni cation, show highest loadings 
on Factor III, production and dissemination. Although Factor I, develofxnent, is 
never highest, it does have modest loadings on at least four of the skill scales, 
with the highest (.50) for producing skills, and the next highest (.45) for 
analyzing skills. These results indicate that although the process skills scales 
are highly intercorrelated (.71 to .90), they do exhibit socne^hat different, if 
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mxeS^ factor pattems. The patterns sem to oake sose "comon sense*' in that 
collecting inforrT'^ticn skills, analyzing skills, and evaluating sklll^ are Dost 
pranlnantlj? associated with Factor il, the evaluation factor; whereas planning, 

producing and concjunfcating tend to be associated with Factor III, production 
and disbenination. None of the six process skills display loadings above .50 
on F<ictor I, develcpcient, but at least four process skills—analyzing, planning, 
producing, /and evaluating- -have loadings above .38. Table 4 below presents the 
analysis of differences between the student and the developer groups on each of 

the scales- 

f 

TABLE 4 

PROFICiEKCY LEVELS FOR STUDEIfTS m FOR DEVELOPERS BY SLBSCALES 



SCALE 


STUDEfiTS 
{N=62) 


DEVELOPERS 
{K=16) 


F 


P 


Field Test and Evaluation Context . . 
DissCTination and Marketing Context. . 


3.99 
3.86 
3.44 


5.39 
5.22 
4.47 


20.12 
19.59 
8.41 


.0001 
.0001 
.0051 


A A' At 

collecting Information Skills 

Planning and Designing Skills 


3.84 
3.55 
3.41 
3.62 
3.69 
4.45 


5.18 
4.94 
4 64 
' 35 
5.O0 
5.55 


19.37 
21.41 
14.58 
13.38 
18.19 
12-24 


.0001 
.0001 
.0005 
.0008 
.0002 
.0011 


*** 

Field Test and Evaluation Cluster. - . 
Dissemination and Marketing Cluster. . 
Publication, Production, and Public. . 
Relations Cluster 


4.04 
3.81 
3.65 
3.40 


5.33 
5.25 
4.86 
4.49 


12.35 
21.66 
12-25 
9.00 


.0011 
.0001 
.0011 
.0039 


TOTAL SCORE 


3.76 


5.03 


19.55 


.0001 
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I!iSTaiJy£!ff REVISIOIJ 
The various data analyses previously described {i.e., cluster analysis and 
factor analysis of scales), designed to deternine if the assured ^'context by 
skills'* matrix organization could be supported empirically, fom^ that though 
the DDiE Diagnostic Instrument iteas were highly intera)rrelated, there were 
at least three we l-defined clusters associated with the three ODSE contexts. 
Though the analyses failed to produce any clusters associated with a single 
process skill (the rows in the jaatrix of Figure 1), the clusters exhibited 
markedly different patterns of process skills- So, although 1) the process skills 
did not emerge cleanly, 2) tavo highly a)rrelated clusters associated with the 
dissesnination context were found, and 3) the itens in the developnent cluster were 
only a srnall (n=9) subset of the 24 itens in the developnent context, the data 
analysis did lend enpirical support retention of the structure of the DD&E matrix. 
Given the preliminary nature of these results and also the fact that ojr sample 
of 78 test Subjects involved only 16 developers, we decided to keep the structure 
of our a priori matrix of three contexts and six process skills, and to focus 
our revision on the refinement of items. It should be noted that most of the 
revisions were based on the cluster analysis results, on the review of appropriate- 
ness for specific cells of the DD&E matrix, and on the levels and distribution of 
proficiency ratings. Only two items were changed on the basis of significance of 

differences between the student and the developer groups. In both cases, the 

3 

Items were rewritten to describe more specific ccxnpetencies. 



Selection of items on the basis of discrimination would be more appropriate for 
a norm i ferenced instrument. The DD&E instrunent is primarily criterion referenced 
Its primary purpose is not to discriminate between groups with different general 
levels of experience, but to identify the levels of proficiency for specific 
competency areas. 
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Re/isix^ns Based on Cluster Analysis 

The fuur clusters were sfaidied to identify itePs which did r.ot cluster 
wiUi other itecss in their colurEn (development, evaluation, or dissemination). 
These items were then reviewed to determine if they should be revised, Eoved 
to a different colunn, or replaced with a new iteii. For example, Iten 13 is 
a developTient item which appeared in a dissesnination cluster. The original 
iten was: 

Take steps to assure that such things as waivers, releases, 
copyright releases, or patent protections are secured when 
appropriate. 

After reviewing the iten itself, and other itens in the cluster, it was de- 
termined that i ten) 13 was (1) describing an activity often considered a disseni- 
nation activity, arid (2) very similar to iteu 56 which was already in the dis- 
semination column. The original iteni 13 was replaced with a developcnent activity: 
Prepare scripts for instructional filcns or sound films trips. 
Another example would be item 45, an evaluation iten which was grouped into 
a dissemination cluster. In its original form the item was worded: 

Prepare an article for publication in a scholarly professional 
journal . 

In this form, the content of the article has not been specified. The item was 

reworded so that it would represent an evaluation activity: 

Prepare an article describing an evaluation study for publi- 
cation in a professional journal. 

Revisions Based on Item Location 

All items were reviewed to determine if the activity they described was 

appropriate for the OD&E Matrix cell in which they had been placed. Items which 

did not clearly describe an activity of their cell were revised or replaced with 
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a new activity stateT^ent. For exaniple, the original activitj described in item 
43 was: 

Detennine the internal consistency reliability of a knowlege 
test instrument. 

Because this item appears in the evaluation process skill row , it should describe 

an activity requiring a critique or review which includes a judgment of value. 

Since the original itesn does not require judgment or decision, it was replaced. 

The new item 43 is: 

Critique a field test plan In tenns of technical adequacy, 
feasibility, and cost effectiveness. 

A list of items in the revised DD&E Diagnostic Instrument can be found on 

the following pages. New or revised items are indicated by an asterisk {*). 
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REVISED DIAGNOSTIC INSTRUHErfT 



Development of Educational Products 

Jjlleetinj In furcation 

*1. Use bibliographic resources such as Research in Education and Current Index 
to Journals in Education to locate information supporting the need to de- 
velop an educational program or product. 

2. Locate existing methods or strategies which can be used in potential product 
development. 

*3. Conduct a search to determine if products exist which could meet your need(s) 
for instructional materials. 

*4. In preparing a proposal for a development project, search for and select 
references which would support the rational and technical approach of the 
project. 

AnatyziTtq 

5- Given a problem statenent, information on the history of the problem, ob- 
jectives and possible solutions, write a 20-page proposal for solving the 
problem, including a rationale for the approach and a development schedule. 

*6. Use theories of instruction and learning to design an educational product 
or program. 

*7. Analyze job or task requirements to determine objectives for an instructional 
program or product. 

8. Categorize instructional objectives in terms of a taxonomy (e.g.. Bloom's 
Taxonomy of the cognitive domain or Gagne's conditions of learning). 

Planniruj 

*9. Plan the budget and schedule for a development project. 

10. In producing specifications for instructional materials determine an appro- 
priate format for the materials. 

llTTrrange learning activities in a sequence to facilitate learning or mastery 
of objectives. *^ 

*12. Design alternative instructional methods for attaining the same set of 
objectives. 



*New or revised item. 
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Producing 

*13. Prepare scripts for instructional films or sound filmstrips. 

*14. Improve curricular materials by revising them according to instructional 
technology principles. 

15. Write exercises which the learner should do in order to master concepts 
or principles in an instructional unit. 

16. Write copy for instructional materials from product specifications. 



Evaluating 

*17. F£'/iew first draft materials to determine if they comply with specifica- 
v'iOns regarding objectives, sequencing, and content. 

*18. Evaluate, alternative methods of presenting instructional material/media 
and instructional methods to match them with instructional objectives and 
learner characteristics. 

*19. Provided with product specifications j the test product, field test reports, 
and expert reviews, make recommendations regarding the nature and extent 
of revisions required. 

*20. Evaluate the feasibility and risks associated with alternative approaches 
for redesigning a product that has failed to meet specifications. 



Communicating 

*21. Prepare a technical report stating the need, rationale, and proposed 
technical approach for devaloping an educational product or program. 

*22. Prepare a memorandum which summarizes the actions taken and problems 
encountered in a meeting with your development team. 

*23. Secure the services of, and meet with consultants to obtain their advice 
and recommendations regarding a development project. 

*24. Make a formal, oralVeport to the administrators of your organization 
regarding the status of a development project. 
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Field Test and Evaluation 

Co ^ Lea ting Inforrsation 

25. Prepare a coding scheme which will be used by a group of coders in pre- 
paring field test data for computer analysis. 

26. Organize statistical data into a meaningful presentation. 

27. Interpret scatter plots. 

28. Design data collection procedures to maintain privacy or confidentiality 
in collecting, processing and storing information. 



Analyzing 

29. List the major factors which jeopardize the internal and external validity 
of a specified evaluation studly. 

30. Formulate significant, answerable questions for an evaluation study. 

31. Analyze discrepancies between expected and actual test outcomes. 

32. Determine if new test instruments need to be developed by reviewing how 
well available tests fit the evaluation objectives. 



Planning 

33. Determine which standard procedure for establishing validity is best for 
your test instrument. 

34. Design a monitoring system that will provide data on the status of the 
operating system (such as actual vs. intended outcomes, unmet needs, 
problems, etc.). 

35. Given a situation where a randomly assigned control group can not be 
established, suggest feasible methods to control for extraneous variables 
that may confuse the evaluation results. 

*36. In planning the evaluation of a new training program, determine the types 
of subjects which should be used. 
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*37. Revise a test instrjr.ent on the basis of data collected durins a pilot 
test of tne instricent. 

5S. V.ake adjustments to test adninistration P'^ocedures when sitoaticnal^^^ 
tors pake such adjustrents essential, in a nanner that will secure and 
protect the validity of the sost inportant data. 

*29. Specify a set of procedures to be followed by r^aluation personnel in 
adninistering test instrument. 

40. Explain a.e inportance of star«dardized procedures in conducting an inter- 
view. 



41. Beternine if tiieoretical assumptions underlying various statistical 
techniques have been violated in analysis of data. 

*^2, Evaluate proposed test instruments to determine if they are valid, re- 
liable and appropriate for specific evaluation purposes. 

*43. Crytique a field test plan in terms of technical a&quacy, feasibility 
•and cost effectiveness. 

*44. Review a draft Yield tesi report to determine if it is correct, complete 
and ready for release.^ 



*45* Prepare an article describing an evaluation study for publication in a 
professional journal. 

*46. Give an oral report of the findings of an evaluation study, 

47. Prepare simple evaluation reports summarizing findings and interpretations 
of collected data, 

*43, Prepare graphs to display numerical information siiimarizing field test 
results. 
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Pi ssgnl nation and Warketlfig 



49. Collect inforT^ation on installation costs for your instnicticnal product. 

50. Design data collection fores for a rarlceting study of an educational 
product. 

*51 . Obtain infonnaticn about strategies used to sarket -ducaticr^l products 
sinilar to yours. 

*52 Retrieve infOTT<ation on politicel, social, and economic factors irfiidi cay 

have a bearing on the dissemination and nsrketing of an educational product. 



53. Identify the crucial characteristics of a target group which may influence 
the dissemination effort. 

54. Evaluate at least three different raarket research techniques applicable 
to a specific development. 

55. Review alternatives for the design of a product In terras of possible 
problems in installing or maintaining tiie product. 

56. Kake arrangements to secure copyrights and copyright clearances where 
needed. 



ol Outline factors which must be considered in disseminating information 
* about an educational product designed for a specific target audience. 

58. Design an effective public 'relations activity for a research and develop- 
ment organization or project. 

59. Plan interviews with potential users for the purpose of determining a 
market for your product. 

< 

60. Prepare specifications for audio-visual materials which will be used 
in the dissemination effort. 
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61. Write hut^^c relaticns scripts for filn aod slide stoitfs. 

62. Writs press releases to disseniitiate information about a new product. 

63- Prepare a sar:ple of an educational product which can be presented to 
a specified target ^roup in order to deterolne if the proposed product 
neets the group's needs. 

*e4. Prepare a brochure for users which describes naintenance and proper 
r usage of an installed prt^duct. 



65. Detemine the thoroughness of distribution Khidi occairred in dissemina- 
tion of an educational product. 

*66. Oeternine the effectiveness of using a dfi^nonstration to disseniinate 
information to your target audience. 

*67. Given the results of a narketing sfaidy, review alternative dissenina- 
tion plans to detennlne vjhich is most appropriate for the predict and 
narket studied. 

68. After product installation, interview users to detemlne if adequate in- 
formation was provided about how to Install and use the product. 



*69. interact with users to help them install a new educational product or 
program. 

70. Deliver an oral presentation to a group of more than 20 professional 
educators to disseulnate' infomation about a project or product. 

*71. Confer v/ith different types of users about their problens in the use of 
and educational product. 

*72. Translate technical information from a niarketing study Into a readable 
suinmary report. 
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APPENDIX A 

REVISED Eiiafr-ponn" ratiijg scale 

ORIGIfJ/U. DIAGMSTIC INSTRmErn" 
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APFBiDIX A 
aE/iSED EiaiT-POniT RATIfJS SCALE 
IJote: Interviews with subjects and analysis of frequency a>unts of levels 
of proficiency reported indicate that some subject., especially the highly 
experienced professionals, needed an eighti; level. The revised jnstrtcDent to be 
used for discrin^inant validation has been codified to include this additional 
statement. 

Level of Proficiency 

1. I have no specific knowledge about this activity nor experience 
with it. 

2. I have read about or seen this activity perfonned, but have ik) 
experience with it and don't really understand it. 

3. I have studied this activity or have frequently seen it perforoed 
and have a good understanding of it, Imt I have not yet done it. 

4. I have a general understanding of this activity and have had socie 
experience with it, ^-^ugh so that I can do it if I have either 
detailed instruction or close supervision. 

5. I have enough experience in perfonning this activity to do it if 
given enough general supervision or general instructions. 

6. I have enough knowledge and experience with this activity so that 
I can perfora this task quite satisfactorily without supervision 
or job aids. 

7. I have had extensive experience with this activity, and can per- 
form it quickly, efficiently, and do a top quality job. 

8. I consider myself an expert in this activity and fully qualified 
to accomplish unusually difficult or cocnpletely novel work. 
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CRIGHiAL DIAGJiOSTIC IfSTHUKEffT 



Eei^elopcent of Educational Products 

1- Retrieve inforaation on political, social or economic 
factors Khich have a bearing on the analysis of an 
educatior^al problem. 

2. Locate existing s:;etfcDds or, strategies which can be used 
in potential product developcent. 

3. Prior to conducting a survey of the literature, prepare 
a search strategy. 

4. In the context of conducting a survey of literature, 
scan and evaluate obtained infonnation for relevance. 



5. Given a problem statement, information on the history of the 
problem, objectives and possible solutions, write a 20-page 
proposal for solving the proble^n, including a rationale for 
the approach and a development schedule- 

• 

6- Provided with current theories of instruction, relate 
thesn- to the formulation of a design for an educational 
product or program. 

7- Provided with field test data on instructional materials, 
examine low gain scores and deternilne if they indicate 
problens in test construction or instructional materials. 

8. Categorize instructional objectives in terms of a 
taxonomy (e.g.. Bloom's Taxonomy of the cognitive 
doniain or Gagne's conditions of learning). 

9, Given funding resources, time and cost estimates, and 
project priorities plan a budget. 

10. In producing specifications for instructional materials 
determine an appropriate format for the materials. 
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11. krrange learning activities in a sequence to facilitate 
learning or nastery of objectives. 



12. Confer with production personnel to deteraine naterials 
needed. 



13. Take steps to assure that such things as waivers, re- 
leases, copyright releases or patent protections are 
secured when appropriate. 



14» Confer with specialists when problenis of production 
cannot be solved on the job. 



15. Write exercises which the learner should do in order 
to caster concepts or principles in an instructional 
unit. 



16. Write copy for instructional naterials froni product 
specifications. 

17. Translate field test data into recommendations for 
action. 

18. Conduct a case study of a program or project. 



19. Provided with product specifications, review a product 
and documentation on product development and field 
testing to determine if the specifications have been 
met. 



20. Informally try out a development product with one or 
only a small group of subjects to observe and record 
how the test subjects use the materials, where and 
why they have difficulties, etc. 

21 . Write a position paper justifying the developcnent of an 
educational product. 
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22. Discuss the development of an educational product with 
jX5tentia1 user-groups. 

23. Interact and contribute in a staff or consultant 
•r-ieeting. 

24. Corrrminicate product specifications to personnel who 
will be producing the product. 
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Field Test and Evaluation 



25. Prepare a coding scher^e which will be used fay a group 
of coders in preparing field test data for cornputer 
analysis. 



26. Organize statistical data into a meaningful presentation* 



27. Interpret scatter plots. 



28. Design data collection procedures to niaintain privacy 
or confidentiality in collecting, processing and 
storing information. 



29. List the niajor factors which jeopardize the internal 
and external validity of a specified evaluation 
study. 

30. Fornnilate significant, answerable questions for an 
evaluation study. 



31. Analyze discrepancies between expected and actual 
test outcomes. 



32. Determine if new test instruments need to be developed 
by reviewing how well available tests fit the evaluation 
objectives. 



33. Determine which standard procedure for establishing 
validity is best for your test instrument. 

34. Design a monitoring system that will provide data on 
the status of the operating systen (such as actual 
vs. intended outcomes, urmet need$, prdbl^s, etc.). 



35. Given a situation where a randomly assigned control 
group can not be established, suggest feasible methods 
to control for extraneous variables that may confuse 
the evaluation results. 
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A-6 



ERIC 



36. Detennine criteria for selection of field test sites 
for a specified educational product. 



37. Kake revisions in test instruinents based on evaluation 
data. 



38. Kake adjusfanents to test administration procedures, Khen 
situational factors niake such adjustasnts essential, in 
a luanner that will secure and protect the validity of 
the nx)st iniportant data. 



39. Prepare a test administration manual. 



40. Explain the importance of standardized procedures 
in conducting an interview. 



41 . Determine if theoretical assumptions underlying 
various statistical techniques have been violated 
in analysis of data. 



42. Evaluate test instruments using data collected in 
try-out and revision cycles in order to reconmend 
instrument revisions for the final field test. 



43. Determine the internal consistency reliability of 
a knowledge test instrument. 



44. Given a report on the evaluation of an educational 
product, identify the purpose of the evaluation 
and the steps used in the process. 



45. Prepare an article for publication in a scholarly 
professional journal. 



46. Give a short speech or oral report* 



47. Prepare simple evaluation reports summarizing 
findings and interpretations of field test data. 



48. Prepare graphs to display nunerical information. 
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Disse^iinatlcn anj Marketing 

45. Collectifig inforriation on installaticn costs for 
your instructional product. 



50. Design data collection foras for a narketing study 
of an educational product. 



51. Use at least three different infonnation services to 
obtain Infornation about educational products similar 
to yours - 



52. Con:.truct an annotated bibliography to accrapany a 
brochure describing the product being disseminated. 



53* Identify the crucial characteristics of a target 
group which may influence the dissemination effort. 



54* Evaluate at least three different market research 
techniques applicable to a specific development. 



55. Review alternatives for the design of a product in 
terms of possible problems in installing or main- 
taining the product* 

56. r-teke arrangements to secure copyrights and copyright 
clearances where needed. 



57. Outline factors which must be considered in disseminating 
information about an educational product designed for a 
specific target group. 



58. Design an effective public relations activity for a 
research and development organization or project* 



59. Plan interviews with potential users for the purpose of 
determining a market for your product* 



60. Prepare specifications for audio-visual materials which 
will be used in the dissemination effort* 
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A-8 



61. Write public relations scripts for film and slide shows. 



62. Write press releases to disseminate infonnation about 
a new product. 



63. Prepare a sample of an educational product which can be 
presented to a specified target group in order to 
determine if the proposed product meets the group's 
needs. 



64. Prepare the specifications for a manual which describes 
maintenance and proper usage of an installed product. 



65. Determine the thoroughness of distribution which occurred 
in dissemination of an educational product. 



66. Carry out an evaluation of the effectiveness of a 
denionstration of an educational product. 



67. Given a marketing study identify: a) the problem, 
b) how the study was designed and c) the outcomes 
of the study. 



68. After product installation, interview users to determine 
if adequate information was provided about how to install 
and use the product. 



69. Interact with users to establish better understanding 
and cooperation in installing a new educational product 
or program. 



70. Deliver an oral presentation to a group of more than. 20 
professional educators to disseminate information about 
a project or product. 

71. Conduct demonstrations on the use of an educational 
product. 

72. Translate quantitative or numerical information from a 
marketing study into verbal or narrative form. 
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APPENDIX B 



' RESULTS OF THE CLUSTER ANALYSIS 



121 

•er!c 



Results of the cluster analysis are presented below. For each 
cluster, the items are listed in the order in whidi they were added to 
the cluster. Tne letter to the left of the original item number in- 
dicates the DD&E matrix coluin in which the iteni was located. D = 
Developrient, E = Evaluation and M = Dissemination/Karketing. 

Cluster i - Publication, Production and Public Relations 

M 60. Prepare ^specifications for audio-visual materials which 
will be used in the dissenination effort. 

M ei. Write public relations scripts for film and slide shows. 



M 62. Write press releases to disseninate information about 
a new product. 



H 56. Make arrangements to secure copyrights and copyright 
clearances where needed. 



13. Take steps to assure that such things as waivers, re- 
leases, copyright releases or patent protections are 
secured when appropriate. 



M 58. Design an effective public relations activity for a 
research and development organization or project. 



H 70. Deliver an oral presentation to a group of more than 20 
professional educators to disseminate information about 
a project or product. 



D 21. Write a position paper justifying the development of an 
educational product. 



D 12. Confer with production personnel to determine materials 
needed. 
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Cljster I (continued) 

E 45- Prepare an article for publicaticn In a scnolarly pro- 
fessional journal. 



D 9- Given funaing resources, tine and cost estfeates, and 
projefct priorities plan a bu^lget. 



D 1. Retrieve infonn;»ticn 
factors Khidi have a 
educational problen. 



on political, social or eccnroic 
bearing on the analysis of an 
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Cluster II - Field Test end Evaluation 

42. Evaluate test iDStnrr;ents using data collected in try- 
out and re/isicn cycles in order to recocn^end instnr:ent 
rei^isicns for the final field test. 



E 37. ?^3ke revisions in test instnrtents based on evaluation 
data. 

D 17. Translate field test data Into recocHjendations for 
action - 

£ 39. Prepare a test adninistration nanual . 

D 7. Provided with field test data on instPictional materials, 
exa^^ine low gain scores and determine if they indicate 
problems in test construction or instructional materials. 

E 32. Beteraine if new test instnCieRts need to be developed 
by reviewing how well available tests fit the evaluation 
objectives. 

D 20. Informal3y try out a developnent product with one or 
only - sjnall group of subjects to observe and record 
how the test subjects use the materials, where and why 
they have diffiojlties, etc. 

E 43. Determine the internal consistency reliability of a 
knowledge test instrument. 

D 6. Provided with current theories of instruction, relate 
then to the formulation of a design for an educational 
product or program. 

E 31. Analyze discrepancies between expected and actual test 
outcomes . 

E 38. Make adjustjuents to test acfcnini strati on procedure?, when 
situational factors make such adjustments essential, in 
a manner that will secure and protect the validity of 
the most important data. 
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Clyster II (ccntlniied) 

E 25. Prepare a coding scher.e which will be used by a group of 
cocers in preparing field test data fcr computer analysis. 



E 33. LeterTiine >riiidi standard procedure for establishing validity 
is best for your test instnir^ent. 



E 47. Prepare sirrple evaluation reports su3:3arizing findint,s 
and interpretations of field test data. 



Design a rionitoring system that will provide data on the 
status of the operatinq systen {such as actual vs. intended 
outcoHies, unxet needs, problems, etc.). 



E 30, FoHTjulate significant, answerable questions for an evalua- 
tion study. 



35. Given a situation whei^ a randi^ly assigned control group 
can not be establisn^, suggest feasible oethods to control 
for extraneous variables tiiat nay conl^jse che evaluation 
results. 



D 18. Conduct a case study of a prograni or project. 



E 28- Design data collection procedures to maintain privacy or 
confidentiality in collecting, processing, and storing 
information. 



E 25. Organize statistical data into a meaningful presentation. 



E 41. Detennine if theoretical assimptions underlying various 
statistical techniques have been violated in analysis of 
data. 



E 27. Interpret s .ter plots. 

E 40. Explain the importance of standardized procedures in con- 
ducting an interview. 
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In tte context of ccnductifif a Sijr*^ey of literature, scan 
and evaluate obtained Inforaaticn for relevance- 



List tr^e najor factors Khidi jeopardize the internal and 
external validity of a specified e/aluation sbjdy. 
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Cluster III - Dissenlnaticn (Plerining and Evaluatlcn) 

Vi 57. Outline factcrs whidJ r.u5t be ccnsidered in dissecninating 
Ififonr'ation about an educational product designed for a 
specific target group. 

T'l £6. Carry cut an evaluaticn of the effectiveness of a dexon- 
straticn of an edmraticnal product. 

H 59. Plan inter/iews with potential users for the purpose of 
deternining a narket for your product. 

*A 55. Review alternatives for the design of a product in tenss 
of possible probleriS in installing or o^aintaining the 
product. 

J5 65. Detem^ine the thoroughness of distribution which ocoirred 
in dissemination of an educational product. 



A 68. After product installation, interview users to deteraine 
if adequate inforaation was provided about how to install 
and use the product. 

E 36. Determine criteria ffir selection of field test sites for 
a-spee^fied educational product. 

M 63 Prepare a sample of an educational product which can be 

presented to a specified target group in -^-der to determine 
If the proposed product meets the group's needs. 

H 54. Evaluate at least three- different market research techniques 
applicable to a specific develop^-ent. 

K 71. Conduct demonstrations on the use of an educational product. 

M 51. Use at least three different information services to obtain 
infornatlon about educational products similar to yours. 
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Cluster III (ccntifjued) 

D 22. Discuss the development of an educational product with 
potential user-groups. 

H 64. Prepare the specifications for a f^anual which describes 
inaintenance and proper usage of an installed product. 

D 24. Connunicate product specifications to personnel \&tO will 
be producing the product. 

D 14. Confer with specialists when profalens of production can- 
not be solved on the job. 

M 69. Interact with users to establish better understanding and 
coi^peration in installing a new educational product or 
prog rani. 

H 67. Given a marketing study identify: a) the pipbleni, b) how 
the study was designed and c) the outconies of the study. 

H 49. Collect infonnation on installation a)sts for your instruc- 
tional product. 

lA 53. Identify the crucial characteristics of a target group 
which may influence the diss^ination effort. 

E 44. Given a report on the evaluation of an educational product, 
identify the purpose of" the evaluation and the steps used 
in the process. 

H 50. Design data collection forms for a marketing study of an 
educational product. ' 

H 52. Construct an annotated bibliography to accompany a brochure 
describing the product being disseminated. 

H 72. Translate quantitative or nunerical infonnation from a 
marketing study into verbal or narrative form. 

D 23. Interact and contribute in a staff or consultant meeting. 
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Cluster IV - Develo^r^nt 

11- Arrange learning activities in a sequence to facilitate 
learning or naster/ of cfejectives. 



15. Write exercises which the learner should do in order to 
naster concepts or principles in an instructional unit. 



19. Provided with product specifications, review a product 
and dooinientation on product developoent and field test- 
ing to deternine if the specifications have been net. 



8. Categorize instructional objectives in teres of a 
taxcnony (e.g.. Bloom's Taxononiy of the cognitive do- 
rain or Gagne*s conditions of leaniing). 



10. In producing specifications for instructional materials 
determine an appropriate format for the materials. 



D 16* Write copy for instructional materials frora product 
specifications. 



D 2. Locate existing methods or strategies which can be used 
in potential product developcnent. 



D 5- Given a problem statement, information on the history of 
the problem, objectives and possible solutions, write a 
20-page proposal for solving the problen, including a 
rationale for the approach and a developinent schedule. 



D 3- Prior to conducting a survey of the literature, prepare 
a search strategy. 

r 
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IteT.s Which P ^ 'd tot Appear in a Cluster 
E 48. Prepare graphs to display ninerical inforrjation. 

£ 46. Give a short speech cr oral report. 
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Development and Validation of DDSS (Developiaent, 
Dissemination, and Evaliiation) Conpetence AssessEjent Battery 



Laird Blac^weli 
Paul D- Hood 
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7h*' l-ar We.st DD&E FmiCtional Cons>etence Training Progran is a corpetence- 
i-ni .--'d prui^rst^ionai development progran whic^ provides training resources (23 
ini*truct lonai nodules) airbed at "entry- level** skills and knowledge required in 
^-ducdtionai developcient, dissenination, and evaluation {!>D&E) . One part of the 
program's effort has focused on the developraent and validation of an assessment 
.ystem which is designed to aid both student and instructor in ' CD progran 
pidnninq, izi progress ronitoring, and (3) exit assessment . Although this assess- 
rmt L^ittt/ry was designed for use with the training program, we see jt as having 
J TXiTfs ^MTitrral utility as a model assessment system for coi:5>etence-based pro- 
fessional training programs and as a flexible and adaptable set of assessioent 
instruments that may be employed in any competence-based curriculum -i^se con- 
tent incly^dQS edbcational product development and evaluation. 

Since the assessment battery is intended to provide the basis for a 
s^-quence of reliable decisions (by students and instructors) about classification 
rather than a one-shot decision about selection, it was felt to be important to 
develop a variety of instruments that would assess similar skills and knowledge 
by different wthods so that decisions could be based on several sources of 
information used simultaneously or sequentially. Although this eii$)hasis on a 
mutti -method assessment system has been maintained throughout development; th^ 
form, scope, and content of the individual assessment instruments have under- 
gone some changes through the test, review, and revision cycles of formative 
evaluation. 

B. PROTOTYPE IHSTRUMENTS 

1. Initial Development and Pilot Testing of Assessment Instruments 

After review of a number of alternatives, three basic assessment devices 
were selected for inclusion in the Competence Assessment Battery: self-ratings 
of competence, knowledge test, and job sample tests. 0ue to time and money 
constraints, it was decided to pilot test the proposed assessment system by devel- 
oping and testing two mini-batteries, each based on the content of one of the 
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.i iT.-truc* 2^: ii d.;]*-' . Tf • * -t.at t v-r could, then, be cakc-rj a*^ 

Stlf-r-iitii,^ ittjn-;, kr.owledqv t^-t:! itc-nis, and job sarnie exercises were 
derived dirtrctl^ frcr^ thi.- otjectives and instructional content of the two selected 
LV'^L zy^4al*-.^. Pilot testing with ess^loyeos in educational de\-elopsent positions 
(i:=2S and for the two nQdulc-i; re^-pect ively) with widely varying levels of 
vxi-c-r ienc- denonstrated the feasibility of the nulti-iaethod battery approach/ 
tfU* l*rft rkiTty qut'Stions r*^arding tht: <ic^ievez3ent of convergent and discriminant 
u-oncurrent validity. Although there were often high intercorrelations between 
iii..trunH.-nts, the restricted content of coc^^etencies in the mini -batter i<5S pro-- 
duc«j?d a homogeneity vniich severely limited the possibility of deiaonstrating 
diocrminant validity- It was decided that future development should cover a 
nuch broader range of competencies and that greater care should be exercised in 
matching the specific content of con^^etencies across instruments. 

In addition to the self-rating items specific to the two instructional 
nodules, a "DDirE Diagnostic Instrument" was developed consisting of a broad 
range of self-rating items derived directly from competencies identified (by 
survey of professionals as described in Hood, Havassy, Lash, and Ward, 1973) 
as ^important for entry-level professionals in any of the areas of educational 
DDfcrE. While it was not feasible to develop knowledge tests or job sample tests 
covering such a broad range of DD&E skills and activities, it was possible to 
develop a broad self-rating instrument, which could be helpful in student program 
p^inniiiv, Vi'? to the relative c-"--" ^nd low cost of development, administration, 
and scoring of this type of assessment instrument. The items on this Diagnostic 
Instrument were organized in a matrix so as to yield information about pro- 
ficiency in three contexts (i.e., DD&E) and in six process skills (i.e., collec- 
ting information, analyzing, planning, producing and implementing, evaluating, 
and communicating). After extensive refinement of items based on criticisms by 
local DDfcrE professionals, the Diagnostic Instrument was administered to 78 students 
and professionals in educational DD&E. Analysis of variance indicated significant 
differences in ratings between students and professionals in the expected direc- 
tion for almost all the items. Although ratings were highly intercorrela*-ed, 

Er|c 133 



xu's "cont^ijxt by skills" ruatrix orqaiuzatiou was given soae support; both cluster 
aiiiiiyri--. and factor dialysis produced three cliisters of itens clearly associated 
with tlic- thre*^- contexts (DD&E) but ratings did not cluster by any one of tlfe six 
process --.kill:.. However/ factor analysis revealed different, neaningful patterns 
ot skills leadings on the three DD&E factors. Cn the basis of this data, sone 
I tens were revised or refined in preparation for the following field test. 
Although ..eparatc skills had not been validated, the natrix was retained as a 
device for rumtaming reasonable unifomity among the three sets of "DDSE context' 
itens in terms of their representation of all six process skilis- 

2. U'\'"lut-rrf"nt of a Product Developirjent COnPetence Assesspent Battery 

in order to focus further effort it was decided to construct and validate 
3ust one major assessrsent battery. l*he area of "product developiaent" was chosen. 
The five DD&£ Training Program instructional modules in this area cover product 
design, product engineering, review, tryout, and revision. Although the modules 
er-phasize orientation and familiarization and general procedures employed in 
product development, and the assessment battery was constructed to reflect the 
general content of the modules; it was decided not to constrain the content 
of the battery to conform specifically to the content of the modules. Though 
focused primarily on development and on entry-level (orientation and familiar- 
ization) competencies, the assessment battery was constructed to be considerably 
broader than the instructional modules both in content and in the level of pro- 
ficiency assessed. 

In addition to the DDSr E Diagnostic Instrument which provided self -rating 
it^zis sampling wide areas of DD&E, a self-* •'rir.^ i-. ' ' -'V* -r ^ 

and a job sample test, each addressing the area of "product development," 
were constructed. The self-rating items were derived from the objectives of 
the modules as well as from competencies related to product development which 
experts agreed were appropriate to entry-level DD6E practitioners. These items 
were similar to those classified as in the "development" context of the DD6E 
Diagnostic Instrument, but were oriented toward somewhat more specific compe- 
tencies and activities. The knowledge test items and job samples were derived 
from: content analysis of the "development" modules, analysis of other R6D 
instructional and evaluation materials, and expert opinion as to the activities 
professional developers would likely encounter. Since the job sample exercises 
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w»-r»- invj^r*2« d # " Jcvvlopri»T*t" skills rather than knowledge about specific 

suM^-ct listt.-r .-oT.t*.nt 2r*>,' i : -.uch skills night he exercised, a subject natter 
.if'-a (the T(j, U^r.dini Library Pru}ect at Far Kest Laboratory) was chosen vtiich 
w.i . J -ir^i*- uni strtiightforwiard as possible and for whidi the all necessar/ 
i..:,ck^^rour.i information could bv acquired easily and quickly. The "Librarian's 
•-tjnual" var. chost^-n the product around which to structure rhe developisent 
activities ir, rhc' lot sar:^le. rnis nanual vas to contain instructions for the 
"lii>rarian"' i^u adult consaunity znecber chosen to conduct the training session), 
teaching parents how to use the library's toys with their children. After con- 
sid^.rcible re^-iew and revision, five job sanple exercises were developed: product 
design, preparation for review, tryout, revision, and outlining a jcrrj^-t for 
an instructional film. Although the "script" exercise did not represent as 
general an activity as did the other four and night not typically be encountered 
by entry-level de^relopers, it was included as an example of how job sanrpie 
exercises night b^ constructed to assess more specific or technical skills. 

The various instruments of the Assessment Battery were intended to be 
redundant; the self-rating items, knowledge questions, and job sasq>le exercises 
were designed to ai^sess many of the same competencies by different methods. 
However, the easier and less e^a^ensive the instrument to develop, administer, 
and evaluate, the Lrc^ader its scope. Thus, the knowledge test covers some know- 
ledge not relevant to the job sample activities, and the self-rating items 
address many conpetencies not covered by either of the other two instruments. 

3. The Formative Field Test of the Competence Assessmen t Battery 

The purpose of this field test was primarily formative, that is, to gain 
information that would prove useful in revising the battery. Because the costs 
of securing suitable test subjects, making arrangements for testing, scoring, 
and analyzing data are relatively high for a battery of this kind, the sample 
was somewhat small but was carefully selected to provide a very broad range of 
relevant experience and training. Thirty-one master's level students from the 
Educational Development Programs of five universities were recruited with the 
help of site coordinators as were 19 professionals in Educational DD&E from 
six R&D agencies. All participants were paid. In addition to this formal test- 
ing, a group of expert reviev/ers were asked to examine the content of the battery. 
Samples of completed test materials, the scoring system, and other pertinent 
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mfomaticr, -*x-r» ^^urplied to r^-vi^-^-r^-rt. in order to help thc-c inake zijt^stantive 
■iTi-^l n^-thr-dological critiqm*r and rc'CcnzK^ndations. 

4. Siscrininatir.n Setveen Students and Profc-ssicnals 
Sc-if-iatirAgs > 

Factor analysis of the self-ratings on both the Diagnostic Instrument and 
tne rore specific product developnent self-rating instrinaent re*,realed 8 sig- 
nificant factors. IVto each related to evaluation and dissenination and four 
rtfiated to developraent. Although the ratings did cluster according to context 
(i.e. DL»6.E),they did not tend to differentiate according to the other dinension 
("Process Skills") on the Diagnostic Instrunent natrix. when testees were class- 
ified into •^entry-level students'* "experienced professionals- (15=6), and 
-intenaediates** {N=25) on the basis of biographic infomation {e.g., past and 
present enpioyiisent , years of DD&E experience, products and pi±>lications) ; self- 
ratings of "professionals" were higher than those of "students" on ail 8 factors. 
Analyses of variance indicated that these differences were significant (P <-05) 
for all but one of these factors. 

Knowledge test , 

rne knowledge test was scored according to= whether all requisite informa- 
tion was included ("fraction" score), and how much breadth and imagipation was 
demonstrated ("scope" score). Although the latter score was somewhat subjective, 
scoring criteria were developed and 85-90% agreement was attained between scores. 
(The developer of the scoring system did all the scoring of field test responses.) 
In addition to thes^ three scores described above for the total knowledge test, 
the same three scores were obtained for several subscales (e.g., tryout, review, 
script, design) which were derived from factor analysis and examination of the 
content areas addressed by the knowledge questions. "Professionals" obtained 
higher scores than did "students" on all subscales and totals. Analyses of 
variance indicated that these differences were significant (P<.01) for tlte 
"pass/fail," "fraction," and "scope" scores for the total test, but were sig- 
nificant for only a few of the subscales (e.o., the specialized area of script 
production. ) 
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"I ^.---:/fu^ i -f^ j-.-^ i -n, " ^tA -rrcT-r-- ^--c^^rc-r vc-r/- al^o obtained frr each 
five :ct -^uir^Iv •I'Xi-rcaisv:- .jit -ir. for the total icb sairpl^ (siiEulaticn) 

ti- ii'iiticr, :-i-v<-raI .-'ut ."cai*-^- c^TiSistency of response with th<- 

sr.forr^r irr. irovid.d, writir.g skill, followir.q directicr*s) vere derived free 
factor ar^tfiyri: ar.d *xarsinalion of the ccrs>eter.cies addressed by the exercises. 
Aaann^ -{.rc s.^iCHiils- obtained hiqher scores than did "st-odents- on all si2:?scales 
ar-d totals.- ATi^slyses of ^^ariance indicated that thtrse differences were significant 
{? .01 ir iTiCiSt cases) for rhe three total scores, for the scores on three of the 
iT.davaduai exercir-es (review, revision, and script), and for the "consistency 
witn inforriiticn provided** ar;d *Toliowiri9 directions" subscalej>. 

Multiple correlation regression anaiya^is . 

With the two extrezae groups of the expericiice-ievel classification (i.e., 
- tudenf and "professional") as the criterion; Lhe total and simscale scores on 
the self-ratings/ knovled^e test, and job sample test accounted for 78% of the 
variance (i.e., multiple R=.S8). The multiple correlation coefficient decreased 
only slightly when either the knowledge test or job sasple test was removed as a 
predictor C^.S? and -85 respectively), and decreased somewhat nore when only 
the self-rating was rerioved (R=.80). Total and subscale scores on each of the 
three in:..trusents correlated highly with "student vs professional" classification 
aro^.r.:^ (rcultiple K's ranged from .65 ^o .77). The patterns of nultiple correla- 
tions wrve the sanie when all three classification groups ("student," "profession- 
air," and "interr^ediates") were considered, though the correlations were lower. 
The self-ratings, knowledge test, and job sample test, taken together then, accounted 
fon4:> of the variance in classification (inultiple R=.67). Th^s corri>lation de- 
reared to .S3 when o^ly the self-ratings were renoved. The patterns of nultiple 
-orr». -5tions described above suggest that each of the instrunents provides power- 
ful discrimination, but that the knowledge test and job saisple t^st nay provide 
somewhat redundant infomation, while self-ratings provide laore independent 
inforraation. * 

r 

♦Ti^di . 'ual coriparisons of combinations of predictors in a stepwise inultiple 
regression support this hypothesis, for the (squared) multiple correlation of 
"Vnowledq*- --^-t plu.^ self-ratings" or "job sample test plus self-ratings," with 
-xcerience classification is significantly different (P^.05) than that of "self- 
rating" alore with experience classification, while the differences between the 
(squared) multiple correlat^ns of "knowledge test plus job san?>ie test" with 
'^xperier^re classification and either test alone with experience classification 
arn not ?:i<)ni ^icant. 137 
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V. CC'riV«-rgv?il or.d DircriiL^r->^:^t vcr.currerit Validity 

£v acs*,'>r.rytT.t -syrt^-ii m wrJch iif^^-rent instruments h^^x^s roric- 

tr-- .-.^r**- trc>r^--tvrc^*-':. t^y diifvr.-'riT laet^iDds, it shciald be- possible to incr^^a^^: 
th^. v<ftii<5ity of cl^iiCF.af icoit 2crA dccisicns vhi<3i mast be jsade relevant to pro- 
fit-ivn^cy on thoi-^^ -or^-etor.cjtes. order fcr this iralti-trai t, nmlti-jaethod 
urr.^K5tnc'iit to be effective, hove-.- r, ccnvirrgerit ar.d discriminant validity of 
tte systCTi should be cl^^net rated- Convergent validity would be demonstrated 
by i;at;?nif leant correlaLiors beti^een different ^aethods of rseasinring the sane 
variafcle- Althoiich very few significant <x>rreiations vere fotmd between indi- 
\nduai I tens on the var 1012s iiiStr^SDents vhich presiinsbly address the sazne corrpe- 
tencier.; when iteris were groiuped into subscales, alsnost ail t^ie correlations 
L^-tveen scores on coig^arable subsc^les (i.e., those relating to the saiae oor^^e- 
tencies or cont^t areas) for the knowledge test and job saisple test were signi- 
ficant {P^.OD. The three total scores {-pass/fail," "fraction, - and "scope-) 
ver^ highly correlated across instruments (the knowledge test/ job sairrple test 
correlations for hese scores were .50, .55, and .52 respectively). Tnere were 
few significant correlations oetween self-rating sasbscales and those of the other 
two instriinents. 

According to Canpbell and Fiske (1959) there are two ways of deiaonstrating 
discrininar.t validity: (1) higher correlations between ineasures of the same 
coi3petence obtained by different rsethods than between ineaszires of different 
conjpetencies obtained by different laethods, and (2) higher correlations between 
measures of the sane cor^tence obtained by different inethods than between 
neastures of different ccapetencies obtained by the sane sethod. There was little 
of the first type ^f evidence for discrininant validity — only in soae cases 
were the "sane competence, different method- correlations higher than the -dif- 
ferent cci:5>etence, different nethod- correlations. There was no evidence of 
the second type to denonstrate discriminant validity - the correlations anong 
subscales on the sane instrument were usually high (R=.30 to .80) and were 
consistently higher than those between corparable subscales on different instru- 
ments. 

so, although there was evidence that the assessment instruiiients do discrim- 
inate (when used independently or fogether) between students and professionals 
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tl^t 1*--^- * the kriowltra^;*- test ar>d ^.eiqilc- tirzt ar^i laeasiariT.g f^c^irifi of 
th*. 00115:^1^7. ci<ro^ thert; i^as latri<j eviconco of di^crisninarjt validity. This 

:ht su^;^*.':.t tr^jit a strorig Racier of ^er^eral ability or irjt^rlliger-ce is acccrat- 
-T,-;; fcr /-'rr/ of th*- vara^ility irt scores. 

•itie *-3q.-ert reviewers had few cosraeijts or criticisrss aiout the fora or 
cuijtt^fit of the s*-:lf-ratirig iteins on either the Diagnostic Instninent or the 

sei^sTHiT^i :ri:itriinerit zx^re specific to product deveiopiaejit.. ^ey did, however • 
ar.dic^lv thdtt the- 8-poirit "prof icier.cy :-ual',-" -*chic±i testees lised to rate their 
pruficien^r/ on tht ooirpetencies specified in the itecis was corfusing in that 
it ccci;in«iK3 levels of knowledge and experience {e.g., "I have read about or 

thi:^ activity perfcnoed, but have no e>:perience with it and really don't 
ur.c«i.t^id it--) Tliough this combination of knowledge and eDq>erience was inten- 
tional # we decided to chani^e the scale by developing a 6-point "level of perfor- 
ranee" ^cale ^hich allows tei,tees to rate their perfonnance capabilities primar- 
ily m terns of varying ajx>unts of supervision {e.g., "1 can perforn this activity 
if I an qiven enough general supervision or general instruction."). 

Since factor analysis and cluster analysis supported the •'context" dis- 
tir.ctions (i.e., developraent, dissemination, and evaluation) in the Diagnostic 
Instruinent natrix, but not the "process skills" categories, it didn't seen 
whorthwhile to retain the natrix organization. Instead, we selects, those items 
ou both self -rating forms wjiich had strong factor loadings and combined them into 
one 6^-iten self-rating form which still focuses primarily on "product develop- 
ment" (including evaluation) cox:5petencies, but which also includes a somewhat 
broddcfr range of DD&E Coripetencies including those related to dissemination. 

Reviewers had few criticisms of the knowledge test. However, on the basis 
of their comnents and on the results of the factor analysis and analysis of 
variance by student vs professional, several of the questions were refined and 
a few were eliminated. We deleted all the questions which required definitions 
of DDirE terms and retained jast the short answer questions involving knowledge 
or application of knowledge in product development and evaluation which v,e felt 
was requisite to performance of the job sac^sles. Suggestions for scoring 
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-;!rtrir."3 niir.ual vhich watt tric-d cut 



^5vl=t r<.-vi"----'--rr ^nd t»ict.r<rs wfrr«? favorably ir^rtrrsed vith the jcb r.cEpl<- 
te.;t. /ipi-rcxir^tvly S5t of tsrstec-s thought th-e exercises rr-gaired iise of rel- 
c-vent educaticr-al frcdiact dc-velcpmc-nt coEpetCT-cieE, and i»,«p.roxiiaately 75% thought 
thc-y wt-rt tir-ful ir. ir-dicatir^ areas-, of str^r.gth a- .d weaV-KC-ss. However, less 
than half tt<-; t-.-stcc-s thcsaght thc-ir pc-rforrsar-ice was ref reseptative of t^t- 
quality of work they cculd do in similar real-life de\-elopiaerit tasks. -J^iis 
trc-bleoa ws? nost freq-uently ^.ttrihuted to shortage of tr&3 to work on the exer- 
cir^-.- the basis of ther.e coinaants ar.d the results of fector analysis ar.d 
aria-'VEis ui variar^ce, we elininated one exercise {"preparation for review") 
ar,d =-hort<^r-e-<? ,-»nd refined the others. Since ccane students corqplained of too 
3ittl«? guidance and sorx? professionals conplained of too rrach, we tried to roake 
rhe instructions a^ clear and inclusive as possible without overly constraining 
possible sol-j^icn-. According to the reviewers and testees, we were reasonably 
successful in cur atter^t to avoid "Far West Laboratory jai^on" in the exercises. 
However, despite efforts to develop a scorinq systen which would be universally 
acceptable and usefux, our scoring criteria and laethods are bound to reflect 
a particular concept of educational developnent. For this reas on and because 
of th^ necc r.c-.a rilv hich subiective scoring required f or ccgplex joh sasgles^ 
we have (in our scoring ganual) attempted only to suggest criteria and nethods 

for scoring the attainiaent of competencies which would allow a reasonably . 

good agreei:i.-nt uaong scorers. have empha sized that supervisors or ecployers^ 
rtni^lH ;^dopt different criteria and/or methods to suit thiir own purposes and 
needs. 

C. VALIDA7IOH OF THE Rg/IS£I> EATTERy 

1. The Validation Field Test 

Sixty-six subjects were recruited fron ten different sites, including 26 
subjects fror four univeristy locations, and 40 subjects fron six educational RSD 
aqenci'--.. The P&D agencies included two educational laboratories, one university- 
based R&D agency, two non-profit a&D organizations, and one govemnsnt RSD agency.* 

*7o avoid a possible bias, none of the validation subjects were Far West Laboratory 
emloyees. 
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j arpo-^ cf th*r vai icid»t icrj i^tudy vas to eaxdore the capai^ality 
,2 tr.'r taXt^-r-i ti oi:--c:ri3T.2? at'- o-.vr <s iiride range of Eil; easpericr.ce, ve 
r T,t-;crt p-r-".?r *--cl cf tv% t-.st^i^- to lOD in recniitir.g test sii^jects vith 

rw^cr^' <"f *r3q.-rivT.cv ir. *-^uvat acr:al s^roduct develcr^it. All participants 



Ir. t*.-rr:ir^ cf d»--gre*? of eax^rience the validation S5i)jects ray be divided 
xntQ thre*.- ^roup-: ^3 si±*3ects with no education E&P work essperience and little 
or no record of authoring any kind of ediscational HSD pii^iication or product; 

.;uov^cts witii rodost vork experience and some record of producing pub- 
iiciticr.F as.d products: and 32 si±^jects vith significant RSD work ea5>erience 
and a -i^st^^Titial record of publication and product developsjent. Since the 
Far -rfest IjI^&E Fir^ctional Co2:3:etence Progran is focused at entry-level corpe- 
t*^nce, which ve define as K.A. degree plus one to two years of experinece^ the 
recond ^-nodest ej^rience"} group is closest to the progras's target population. 
Tiie other tvo group^s provide a -bracket- which help to define a broad range of 

indicated in Table 1, there are rajor differences asiong the qzovps 
an aqe, in year- of H&D work e^sr^rience, in nunber of publications and products 
irr.duced, and in level of education.* Additionally, the high R&D e^q^erience 
grou:> is predoninantiy mle; substantially nore of the ipodest R&D e^rience 
groi:p have ^^cnjC teaching experience. 

As we see in the next section, these three groups also differ markedly 
in tenns of their self -appraisal of R&D conpetencies. 



♦The 3»dest R&D experience group includes three doctoral level subjects. i-?o 
w^-e in acadenic departments; the third was a recent eriployee of an R&D agency- 
B^'^^usA of their relative lack of significant R&D work experience and very 
rodest record of publications and products, they reseribled the modest R&D 
experience group nore than the high experience group. 
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TABLE 81 

BlOtSyrrilC DATA m THE VALICATICIJ GROUPS 





RSD E33>£RIO;CE fiid PRODUCTIVITY 


£. 1 


fJ^e 


riOucSc 




Ulsrber in greig? ^ 


13 


21 


32 


Age (years) 


24.4 


30.3 


38.4 


Percent na^' 


3\% 


ZB% 


84S 


Percent haye taught 


62% 


sn 


69S 


Years R5D Kork e^cperience 


0.2 


1-4 


10.1 


JJr. articles in professional journals 


0.1 


0.7 


2.6 


!lr. teGhnicai RiO reports 


0.0 


0.2 


8.8 


Hr. evaluation reports 


0.1 


0.6 


3.6 


Ur. chapters in professional books 


0.0 


0.0 


0.7 


Kr. other publications 


0.1 


0.7 


1.9 


IJ** ti*^ •"^/AccpQ^THpnt instrin&nts 


0.0 


1.1 


8.5 


Nr. progroiraed lessons 


0 "2 


0.9 


8.1 


lir. slide tapes/audio tapes 


0.0 


0.5 


2.1 ■ 


Hr. instructional filnis/T.V. 


0.0 


0.6 


0.3 


Jlr. other a&D products 


0.0 


2.3 


2.3 


Percent with PhD's or EdD's 


02 


U% 


37% 
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m thv fi.>ld test of ti^LC version of the self-rat icg ir^tnsaerit, we 
i nirj lifi<;d ..ix 3ev..l p..rfo«nar.ce scale ^hidi omits all refcrer.ccs 
to knr^lc-eqc- ar.d fc-ojs.-<: r^lc-ly cr; porfon=2r.ce.* ll^e scale is as followsr 



L&vel of Ffcrfonaance 

1. I ciirjiot perfora this activity even -^ ch supervision 
c.r <3i3idaiice- 

2. I can perfor^i this activity if I have either detailed 
instruction or close si^rvision- 

J. I can perforin this activity if I am given enough 
general supervision or general instructions. 

4. I can perfona this activity quite satisfactorily 
without supervision or job aids. 

5. I can perform this activity quickly and efficiently 
and can do a top quality job. 

I consider rr/self an esgsert in this activity and can 
accoc:.lish unusually difficult or cos?>letely novel w>rk. 



1 



me PJMJ&E Activities Instrument consists of six scales, each containing 
ten itezL^. Tn^ Activities Instruiuent, although shorter in length than the 

Diagnostic Instrunent, covers a broader range of cor5>etencies including^ 
il) Research and Statistics, C2) Evaluation, (3) Instructional Product 
Dr,velopi=ent, (4J Planning and Tecm^ical Writing, (5) Dissemnation, and 
(6) RSD xanagerient- 



.Please co^^are this" ,e~c| scale^th the 

^:rrr™.n! a?;" ri;iir4rsLnd.. ^-.^^^^-t^^- 

haven't ^one it) have been cortoined into the rough equivalent °f 

level 1 (can not oerforn this activity even with ^"P^"'^"^" ^"f f i ^^ei 2- 

ProacieL/ scale* level 4 corresponds approximately to perfornanc^ scale level 2. 

!™i2a^^e^ieLrr^^^^^^^^ 

;^SrS;~: a:otL^coSounding .n^.ledge wi^ S^Se^^/p^^L" 
Spropriate for use with advanced professionals, we consider the 7 point pro 
ficiency scale to be Dore useful for training progran assessment. 
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Table 32 r*r|>orts th« item ar,a scale iseans for the three grcut-s;, and the 
P-levc*ls Latiifd oii t>r^e vay analysis of ^^ariance tests of the difference anong 
tht^ E<arji* for t.^t.- thrc-e rS^D *^3a>fcrience level groups. Table B3 reports the 
scale int<irct>rrelatiori£., CH^nii* -standard deviations, and alpha coefficients 
for the six r^alci?. 

It^ data in Table B2 indicate that all six scales discrirdnate aiaong the 
three qrc^pz, with f -levels {substantiallyi less than .01 or all scales eaarept 
Research and Statistics^ where the P-level is less than .05- ^e student 
group with no hJ^I> work experience {but with sone trainitsg display average 
scale ratings close to or slightly above 3-0 on all scales ("can perform if 
given enoiagh general supervision or general instructionsL*") .* By contrast, the 
high P^D experieiice group has average scale ratings above 4.0 CI can perfona 
this activity quite satisfactorily without supervision or job aids") on four 
scales and above 3.7 on the other two {Research S Statistics, Dissemination). 
TnQ group with loodest RfirO experience has Scale average ratings that are inter- 
nediate on all six scales; however, the nodest R&D e3«perience group is only 
slightly higher th-an the no experience group on the Dissemination and the 
R&D l^^nagenent scales. 

It should be noted that the itens appearing on the six self-rating scales 
have not been selected on the basis of group discrimination; but rather for 
their coverage of RDD&E content that seemed atppropriate for entry level pro- 
fessionals. Given very large differences in ?^ork e^^rience and productivity 
indicated m Table 1, the general character of the self-rating results reported 
in Table 2 should be expected. For instance, note that the high R&D es^rience 
group has average ratings that are substantially higher than the other two 
groups on the R&D Management scale- In this case there are significant differ- 
ences among the groups on all ten items on this scale. By contrast, significant 



^pically. the students who comprised this no R&D work experience group had 
completed one course in research design, one course in tests and measures, one 
course in instructional technology and one or two courses in statistics, -toout 
half had also completed a course in evaluation. 
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TABLE 32. R0DXE ACTIVITIES ITEM KEfiliS 



1 . RESEAfiCH AJiD STATISTICS 





MD Ex 


Qerience ] 








f— 

to 




Gl 
> 




o 


MODE 


HIGH 


O 

_s 
i 


Choose (or design) appropriate statistical 
techniques for data analysis- 


3.08 


2.81 


3.53 


lib 


Use and interpret statistical regression 
tediniques 


2.46 


2.76 


3.38 


lib 


identify the types of questions whidi can 
and cannot be answered by different research 
designs. 


2.69 


2.95 


3.81 


.01 


Design data collection procedures to main- 
tain privacy or confidentiality in collecting, 
processing and storing information. 


3.08 


3 86 


3.97 


.03 


Identify and articulate the problem in a re- 
search study. 


3.46 ' 


4.00 


4.66 


.01 


Formulate testable hypotheses in a research 
study- 


3.31 


3.95 


4.44 


.02 ' 


Prepare a coding scheme which will be used by 
^ group of coders in preparing field test data 
for computer analysis. 


2.69 


3.38; 


3.65 


MC 
lib 


Identify factors which can jeopardize inter- 

«%3 1 ^nA OY^oyTi;^1 t/a 1 T Hi f'V 

nd i ofiu cAUciiias vuifuicjr. « 


3.38 


3.71 


4.00 


NS 


Employ sampling theory and techniques to 
develop a stratified or clustered sampling 
plan. 


2.92 


3.05 


3.31 


HS 


Determine an experimental design and statis- 
tical techniques whose underlying theoretical 
assumptions are consistent. 


3.15 ' 


2-86 


3.56 


NS 


TOTALS 


30.2 


33.3 


38.3 


.05 
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TABLE 32 (Continued) 



2. EVALUATIOH 







RXO Experience 








NONE 


MODEST 


HIGH 


P-LEVEL 


4 • 


fipvi^p A t^st instrument on the bflSis of 
data collected during a pilot test of the 
instrument. 


3.00 


3.71 


4.41 


.0] 


0. 


Design a nwnitoring systen that will provide 
data on the status of the operating systea . 
(such as actual vs. intended outcccies, un- 
inet needs, profalesnis, 6tc.). 


2.85 


2.71 


4.25 


.01 


16. 


Analyze discrepancies betoceen expected 
and actual test outcoines. 


3.31 


3.33 


3-91 


.05 


18. 


Determine if new test instri^nents need to be 
developed by reviewing how well available test 
fit the evaluation objectives. 


3.08 


3.57 


4.06 


.or 


21. 


Given the sunmary report of an initial try- 
out, suggest new niethods to use in the second 
tryout to improve tryout validity. 


3.08 


'3.19 


4.41 


.01 


on 
JU. 


T\i^r\r\4r\n pt/ai fi;)i'inn nf ^ npw traininQ 
program, determine the types of subjects which 
should be used. 


3.38 


3.71 


4.28 


.02 


44. 


Provided with product specifications, the test 
product, field test reports, and expert reviews 
m:ii/-o y^prYifTYnanri a 1*1011^ rpoardino the ndture and 
extent of revisions required. 


i. lb 




A OH 


m 


49. 


Design jdata collection forms for a marketing 
study of an educational product. 


2.92 


2.95 


3.50 


NS. 


50.. 


Prepare a summary report of the test of your 
instructional product. 


3.31 


3.67 


4.34 


.01 


57. 


Specify a set of procedures to be fql lowed 
by evaluation personnel in administering test 
instruments. 


3.00 


3.62 


4.19 


.01 




TOTALS 


31.3 


34.3 


41.6 


.0, 
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3. irJSTRUCTIClJAL PRODUCT DEVELGPKEliT 







R&D Experience 








3N.0N 


MODEST 


HIGH 


_j 
> 

> 

I 

C 


7'. 


Fronj a general description or a proouct s 
purposes, specify appropriate tenainal ob- 
jectives. 


3.69 


1 

4.14 


4.34 


»s 


8. 


Prepare a san^le of an educational proauci: 
which can be presented to a specified target 
group in order to detenuine if the proposed 
product rseets the group's needs. 


2.92 


3.90 


4.41 


.01 - 


-26. 


Given an educational product which you are 
developing, prepare tests and measures to 
assess student perfonnance. 


2.84 


3.29 


3.38 


NS 


28. 


Convert a chapter in an instructional text 
into a branching programmed instruction 
sequence. 


3.38 


3.71 


4.28 


.02 


34. 


With production personnel, determine tne se 
quence of activities for production of instruc- 
tional fnaterials. 


3.08 


3.76 


4.53 


.01 


i 

40. 


Categorize instructional objectives in tenns 
of a taxonomy (e.g.. Bloom's Taxonomy of the 
cognitive 'domain or Gagne's conditions of 
learning).'* 


3.69 


4.24 


3.84 


\\S 


46. 


Write exercises which the learner should do 
in order to master concepts or principles in 
an instructional unit. 


3.46 


4.33 


4.25 


.05 


47. 


Evaluate alternative media and instructional 
methods to match them with instructional ob- 
jectives and learner characteristics. 


3.23 


3.90 


4.12 


.05 

» 


58. 


Analyze job or task requirements to determine 
objectives for an instructional program or 

pi UUUi« L. 


3.53 


3.81 


4.38 


.02 


59. 


Detemine questions you would like to have 
answered by the data collected in a tryout of 
your product. 


3.69 


4.T4 


4.53 


.02 




TOTALS 


J 
33:2 


38.8 

1 

1 r 


41.5 

J 1 


.01 



117 



ERIC 



B-16 



TtiuiZ Z2 (Ccnti.iueu) 
4. PLAIlIIinG a TECHNICAL HRITHJG. 



4. Use bibliographic resources such as Research 
in Education and Current Index to Journals in 
Education to locate information supporting the 
need to develop an educational program or 
product. 

9. Review first draft materials to determine if 
they comply with specifications regarding 
objectives, sequencing^ and content. 

14. Conduct a search to determine if products 
exist which could meet your need(s) for in- 
materials. 

31- Mrit^^a detailed product description including 
all the necessary elements. 

32. Prepare 'a product design which is consistent 
with the stated overall purpose and use of 
the product. ^ 

38. Locate existing methods or strategies which 
can be used in potential product development. 

39. Write scripts for instructional films, video- 
tapes, or -sound fllmstrips. 

"51. In preparing a proposal for a development 
project, search for and~§?lect references 
which support the rational and technical 
approach of the project, 

52. Determine revisions that can be made within 
specified time constraints. 

53.. Review the relevant research literature for . 
evidence to support the selection of an in- 
structional strategy. 



TOTALS 



R&D Experience 



o 



111 
o 



""^ — I 

4.00 I 4.24 



3.69 

3.31 

3.15 : 
2.9a 

3.46 
2.77 
3.46 

3.15 
3.77 

33.7 



4.33 

3.57 

3.48 
3.10 

3.57 
3.52 
3.81 



CD 

a: 



4.55 



4.25 



4.28 



> 



f 

a. 



NS 



4.75 8 .01 



.01 



.01 



4.00 I .01 

4.25 I .01 
3.62 I NS 
4.66 I .01 



3.38 



4.33 



37.3 



4.19 



4.59 



43.2 



.01 



.04 



.01 
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T/iBLE 22 (Continued) 



5. i4zwi:imzn 



IS. Confer witli different types of users about 
their problems in the use of an educational 
product. 

25. Make arrangements, to secure copyrights and 
copyright clearances where needed. 

29. Determine the thoroughness of distribution 
which occurr:ed in dissemination of an educa- 
tional product. 

35. Deliver an oral presentation to a group of 
iTiore than 20 professional educators to dissem- 
inate infonnation about a project or product. 

37. Plan interviews with potential users for the 
purpose of determining a market for your 
product. 

41. Collect infonnation -ort installation costs for 
your instructional product. 

43. Identify the crucial. characteristics of a 

target group which may influence the dissemi- 
nation effort- 

54. Determine the effectiveness of using a demon- 
stration to disseminate information to your 
target audience. 

56. Evaluate at least three different market re- 
search techniques applicable to a specific 
development. 

60. Obtain information about strategies used to 
market educational products similair to yours ^ 



TOTALS 



RSD Experience 



o 



3.62 



3.07 



2.77 



3.08 



3.46 



3.08 



2.92 



3J5 



2.46 



3.31 



to 



4.09 



2.62 



2.71 



3.76 



3.57 



2.81 



3.29 



3.10 



2.19 



3.43 



31.2 |31.6 



4.50 



2.94 



3.38 



4.53 



4.16 



3.56 



3.62 



3.97 



2.81 



3.69 



37.2 
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6. R&D FJUIAGEKBJT 



2. After product Installation, interview, 
users to detennine if adequate infonoation 
•was provided about how to install and use the 
product- 

6. Supervise professional R&D personnel. 

10. Derive a set of questions which could be 

sent with your product to reviewers in order 
to focus their review on critical aspects of 
your product. 

17. Write and submit proposals to obtain funding 
and to negotiate with funding agencies. 

20. Determine financial resources necessary to 

conduct a program or project and use accounting 
procedures to operate within a program or pro- 
ject budget. 

22, Given a problem statement, information on the 
history of the problem, objectives and possible 
solutions, write a 20-page proposal for solving 
- the problem, including a rationale for the ap- 
proach and a development schedule. 

35. Review a draft field test report to detennine 
if it is correct, complete and ready for re- 
lease. 

45. Prepare a memorandum which summarizes the ac- 
tions taken and problems encountered in a 
meeting v/fth your development team. 

48. Evaluate the feasibility and risks associated 
v/ith alternative approaches for redesigning 
a product that has failed to meet specifica- 
tions. 

55. Use management and planning systems such as 

PERT (Program Evaluation and Review Technique), 
PPBS (Program Planning Budgeting System), or 
Critical Path Analysis. 

TOTALS 

150 



RSD Experience 



CO 

o i Q 



3.62 i ^.85 



2.33 : 2.95 



3.62 3.76 



2.92 2.48 



2.53 



3.00 



3.84 



2.92 



2.76 



30.5 



2.38 



2.92 , 3.00 



3.29 



4.38 



2.90 



2.38 



4.44 

4.47 
4.34 



4-38 
3-81 



4.38 



4.34 



4.81 



3.81 



3.16 



31.4 



41.9 



I 

Cu 



.02 

-01 
.02 



.01 
-01 

.01 



.01 



.01 



.01 



.02 



.01 
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qioui^ differencr*^. are least evid^t for the Research and Statistics scale, 
v.^jXK- six of th<? ten items display no significant differences.* 

Turnir.g r^-t to Table 3^ we TOte that there are saoderately hi^ correlations 
ciT^T-q the six scales with correlation coefficients ranging frcia .40 to .82. 
Thi; i^i&iiarch and Statistics scale, is (not siarprisingly) strongly correlated 
1.^9} with the Evaluation scale; but otherwise displays the lowest correlatiojis 
with tht remaining four scales (.40 to .51). Instructional Product Development 
is. the next rest independent scale with correlations ranging froa .51 (with 
sT^^ardH' and Statistics} to .74 ^(with Planning amd Technical Writing). By 
ij^^:itt^:*t, the Evaluation scale shows fDoderat^ely high correlations with all five 
*^th«^r :>cales. It's relatively high (.77) correlation with the lianageaent scale 
hf^ t^^i ily uiiderstood when one exanines the H&D Kangeaaent scale iten content 
tZahl *2) and realizes tl^t anciny of these items have evaluative elesnents. Ihe 
Kar^agtszsent scale, lihe the Evaluation scale, also displays laoderately strong 
correlations with all the scales; the highest being .82 with, Dissezoination ** 
and the lowest being .51 with Research and Statistics. 

■ 3- 7ob Knowledge Questions 

Ihis instrument contains ten short answer essay questions. It has a one 
hour tine linit. Tne ten questions involved tne following. C^e content of the 
following i-> sonetimes 'abbreviated; the ccinplete questions are listed in Jappen- 
dix O 

1. Identify isijportant information aissing in a product 
description. 

2. Specify appropriate terminal objectives for a briefly 
described product. 



*This result is possibly attributable to two factors. First, as is indicated by 
the standard deviaticni> in Table 3, there is siibstantially greater variability for 
this scale. Second, our experienced P&L test siabjects were recruited entirely 

ro-, oda ^ati^nal develupaent agenciei>. T.ie na jarity of tliese subjects vere devel- 
. y^t^ 'jkUiGZ than researchers. Thus, when we checked the biographic data we found 
^^«t, altl.ough tJie nore experienced subjects tended to have taken core courses in 
research design and in statistics, there were nc significant differences arsong 
tiie three groups in terns of the nuriber of these courses taken. 

**The unusually high correlation between Dissemination and Management is partly 
'seen in the group means for these two scales. Generally Jieither the "No R&D Ex- 
perience" nor the "^5odest R&D Experience" groups rated themselves very high on 
either the Manageiaent or the Dissemination ^cales, but the "High R&D Esqperience" 
grou|* generally rated themselves (relatively) much higher on both scales. 
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PXDSE ACTIVITIES SCALE nJTERCCRRELATiOSS. JSAJJS (K) 
STAIJCARD DEyiATiCIJS (SD) MD ALPHA !NI£R?iAL 
CiSiSISTBJCy CCEFFICIEfiTS (A, 



— 4; 

Scale 1 

1 




SD 


A 1 
i 


RSD 
1 


EVAL 
2 


DEV 
3 


4 


DIS 
5 


»ST 
6 


I. 


1 

Research and Statistics \, 


35.14 


10.62 


I 

.95 




.79 


.51 


.41 


.49 


.51 


2. 


^ valuation | 




8.31 


.93 


.79 




.71 


.66 


.68 


.77 1 


3. 


Instnjctional Product ] 
n»velopaent 1 


39.09 


8.36 


.91 


.51 


.71 




.74 


.64 


• 

.58 


4. 


Planning and Technical j 
Kriting j 


39.44 


8.17, 


.93 


.41 


.56 


.74 




.78 


.75 i 


5. 


Dissensination ] 


34.20 


7.08 


.90 


.40 


.68 


.64 


.78 




.82 


5. 


1 

RJD Kanagensent ; 

j 


36.33 


8.48 


.91 i 


.51 


.77 


.58 


.75 


.82 


\ 

\ 
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!• i^j^lyz^^ and ^in^ciiii:; « ^ropoin^ P»di5Ct try^-iat in terms cf the 
r*^l'*vam>.* ai,i i3:5-ortance of tise inforaiation it ii^^ild yield. 

4- Lirt thr*^- cf the iosic criteria for jaadgiiig the adequacy 

r-t.^ji^'ioT^ «^hicfj r?..v*^e bei^ 2i;ade in [anyl product. 

5- Cor5.*arc- the advantages and disadvantages of soiand Eovies and 
rjDvr^d filjsrstrij:^: as instn^ctionai jaedia considered froia the 
vicitfpoint of the ^evelc>per^ the distributor, and the user- 
s' Indicate three questions a developer jai^t vant to ans^wer in a 

tryout about student satisfaction vith a prograrnied text. 

7, Identify the infotsafiition that couid be obtained fron a cx>nsultant 
reviei* of a product that mght not be cbtained in a field test ' 
of the product. 

List tliree questions (for each of three speciiEied types of re- 
viewers! vhich should be asked about a [specifi«33 product- 

9- Identic factors other than the product itself, vhi<A might 
account for difficulty subjects laay encounter in using a [any] 
product effectively. 

ib. Outline the sequence of rjajor steps that are cost in^ortant in 
the production of a sound filnstrip between the two stages (a) 
definition of the instructional cbjectives and Cb) collections 
of the first pilot test data. 

Each iten was scored three ways: (1) pass-fail (?-F), scored 1 or 2; 
C2) fraction (F) of essential elenents covered in the response, scored 0-9; and 
(i) scope iSC) of applicatx**n of knowledge going beyond essential requirements, 
scored 1-3*. Ihe pass- fail (P-F) score was scored 2 (pass} if, and only if, all 
essential eleinents of the answer specified in the scoring saanual were present 
Cequal to a score of Z on the 0-9 fraction score). It is not surprising then 
th.2t the average intercorrelation of these two STOres, averaged over all ten 
xtens (with averages based on Z-transfonaations) is .87. ^e scope score, 
which Ejay be considered a ^bonus" score for responses whidi reveal a -scope" of 
application of knowledge going beyond the specific essential requireiaents set 
for the question, was scored 1-5. Conceptually, the scope score is considered 
to be independent of the P-F and fraction score. A student could fail the item 
and still receive a 2, 3, 4, or 5 on scope, although one might expect that core 
kncrwledgeable subjects would receive higher carks on all three scores. l!he 
average intercorrelation of the scope score with the fraction score was .48; 
with the P-F scfore it was .44. These correlations are positive and significantly 
different froa zero, but they do indicate that the scope score does contain 

*See Appendix D for conplete scoring instructions. 
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srifir.nsatJSTi rAt i-tcs^A iii tfce ctJ^r tvo scores. Althousli there are significant 
ir.f.-rcorr«l43tioni, ancrvg ti.e tbree scores based on the sane q-^&sticeis, the ixitsT- 
c-'rrvl■iXi^-^^'t■ -^"r^r.-^ tL'. t«r:- guei:titT.r-. iMJrc s-arj-risingly small, vith only le* 

ccrrel-at i<^i values above .2S IP < .&5). A factor analysis of the SO 
scores reveai*rd tii^t therv «ere F-.-rhaj:^ as sany factors as there vere questions. 
^Il^rcntly the kr^ledge required to deal with these ten Job Kr^iedge Questions 
m Ejodc-rately specific to each itcsn. Ihere are'raodest relations, for instance, 
an&ng the several questiorii; dealing with review or tryout, and there is a tendency 
for fraction or sccpe scores of ^.e question to be related to scope scores of 
ctii..-r .5u.^tiM^, but none of these relationships are particularly strong. 

Ke now tarn to 45ie question of whether the Job Knowledge Questions scores 
ar.-- capable of discriminating aoong the three R£D work escperience groups. On 
txicining the aten neans for the three groups we noted that both the fraction 

the scope score means were hi^.er for the two grc^ with RSD work e35>erience 
than for the group with no SfiD es^erience on ail ten questions. But anal/sis 
of varian-e ir^icated that only four of the ten questions had score riean dif- 
ferences that were significant at the .05 level. However the total" scores 
(suEcaed over th- 10 itcas) are all significant. See •rableB4. 

JJote that the nodest RSD experience group has slightly higher- total 
scores than the high experience group (all non-significant). Both of these 
esqierienced groups have higher scores than the no e^erience group. 

Ihe pass-fail scores are troubling, primarily because they indicate that 
even the laore esq^erienced groups were averaging only 6.9 and 6.7 questions 
"passed" out of the ten. We have two conjectures; first, the one hour tiiie 
linit is probably too short to pemit a knowledgeable subject to answer all ten 
questions; second, the scoring standards are perhaps unreaiistically high, 
we would recoriaend that the user adapt this test possibly by both reducing the 
nunber of questions and by reLixing the pass-fail criteria. 

rnc fraction scores are based on a 0-9 scale for each item which translates 
to a 0-90 possible range for the totals. Table B4 indicates (in parentheses) 
what the F scores nean in tenns of percentage of the possible score of 90, 
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KEHJS, F-EATiOS MJ) ?-i£VElS fOR THE 
F, P-F, Md St TOTAL SOJRES OIJ JD3 KTJOKLEDGE QUESTIGISS 





RiD EyS>ERlBi££ GiOUP 












mm 


F-Ratio 


?-Level 


Pass-Fail (P-F) 
{P-F as ? possibl^ 


15.3 
(53S) 


16.9 
(692) 


16.7 
(672) 


3.81 


0 027 


Fraction (F) 

(F as " possible) 


70.6 
(78S) 


79.5 
(88S) 


77.9 
(872) 


4.89 


0.011 


Scope (SC) 

(SC as % possible 


25.2 
(SIS) 


31.7 
(632) 


29.8 
(^92) 


7.54 


0.002 



loo 

B-24 



#-3q-^-ii*n*ce4 .,Jt)*-s^ av*^T^%*-^5 j-roxiniatfily tr^i percentage points higher. The 

aT:»^:^-ri«-2.r* i ir^^^i^ itj l^.rni. -.f tii^^ir *J:>iiity to deal vith the questions iii 

ir^t ai^j^i/ i'^r^.i^i^-i^v ^Qiii^ Inr/^cA th^i i^r-e-estahiished Tdnismsrx renuirexDents, 

'Ttif J«X Knowledge Sh-ectioins test ..does discriminate between those vith 
i'i^h «ork •-xpericnce and those with s<me experience, ^e differences are 
r.^jX iar^*-# L-Jt they axe statistically , significant. Eovever, there is no 
evi3*^c*-- that those with si±rstantially rore e35>erience perforsi better on this 
t^s.n^^, iT3 fact th- scores of the hiqh <>3cperience grov:p are slightly lover than 
thor^e of tht- nodest eaa>erience groiip. three total scx>res have siibstantial 

xri-:-rcorreiatxcn {.s^ between F and P-F; .?3 between F and SC; and .71 between 
I -F and SC) - Intenrorrelation of ccoxgs based on each item indicate that 
-siiO ten iterts arc not highly intercorreiated, indeed c«ly 16% of the between 
qasestions score intercorrelations were significant and none exceeded -SD. 

4. J^i Exercise in Bdiacational !>evelopaent 

Ohis simjlation test consists of four related tasjcs. Task 1 is concerned 
witTi preparing guidelines for the develcpcent of a sinrple product (a section of 
a toy librarian's Ejanual for parent training). Task 2 deals with preparing for 
the tryout of the parent training resources )dLt. Task 3 requires responses to 
tTie tryout evaluation report by critiquing the tryout and recoi^aending revisions 
m the product and in the tryout procedure. Task 4 requires a rough outline 
tor a sound fiisistrip script. Following the sasie rationale used in scoring the 
Job Knr-wiedge Questions, each task was scored for pass- fail (P-F) , fraction (F) 
and scope (SC) according to instructions contained in the scoring Eianual. Each 
task was also scored in terns of the extent to which the simulation directions 
(DIR) were followed. Kence there are four scores that are of the sane type for 
4^ac5i task CP-F, F, SC and DIH). In addition to these 16 scores, tasks are also 
scored on a total of 11 specific elenents (e.g., quality of writing of the guide- 
line produced in Task 1, the quality of the tryout questions listed in Task 2, 
the suggested revisions of Task 3, the handling of visual elenents in the script 
outline of Task 4). 
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7-..- .'7 ■ r-- of •-.inruLatioii vx.-rciL'<» vere intercorrcLcted snd tiiea 
I ur^i/r. • ti.-I i-.-T. th..- ca.^.- tU- kr^owledse test, there «erc jaoderate 
.J I .1 !i.'.-rc>-.rr''lJt3t:.»s several scores based on the saiae task, 

; V r^ll. r rr. i^ti-nc iTK-.q scores fcr different tasks. iJowever, in contrast 
» • ^lii z,T^.y-.TtU^ Of Significant int-^rcorrelations found in the knowledge 
f Jlly uulf of the between tasks correlations were significant in the 
-;ir:jl3»ivn t._.-.t. A factor analysis indicafsd that the first principal axis 
t...-tc.r ac-.^cA.--^ for 34t of the trace (total covariance) and that six factors, 
s--..-i:.tir.^.for a total of ^A^ of the trace, had eigenroots above 1.0. ^ese- 
r .It ir^icat. that the siznolaticn test is factorially corsiex, but that there 
1 - .s r^.-i--.'' "^-i.--ral 5.rsn$>et<ince- factor that rtais through it.* 

Ever-/ of the 27 scores displays a significant loading on (i.e., 
c..rr..l..tion with) the first principal axis,- however these loadings ra.nge froB 
. i3 to .HI. Thus, the scores e^diibit varying degrees of association with this 

i.?ral V-rfonnance factor, vrhen the six factors are rotated to a varimax 
solution, (sei^^ T^leBS), discover that every score in Task 1 displays a 
.ignxficant loading on Factor 1- (However, tha Task 1 scope score and the guide- 
line instruction-! score also show substantial loadings on Factor 6, and Task 1 
i-y r.^,rr- has a rodest loading on Factor 2J The only score loading significantly 

Kictor 1 and not in Task 1 is the scope score on Task 2- Hence, Factor 1 is 
alnc.-.>t exclusively identified with Task 1 (Preparing Guidelines) . 

The scores on Task 2 (Preparations for Tryout) are s^lit priinarily between 
Factors 2 and 3; but the scope score on Task 2 has ^dest :loadings on four of the 
.ix factors. Factor 2 is best identified by the "specific-score for specifica- 
tion of test sites and subjects. Factor 3 is identified *y the "specific" score 

^is factor analysis contrasts with the knowledge test ^^^^ ""fc^unt- 

trac. was extracted by the first principal axis and where ^^JrHS^aSd 73% 
inq for 81t of the trace, had eigenroots above 1.0. Exght f^^^^Sf sinu- 

tho trace. Hence the knowledge test required two tore factors ^J^^^^^ 
lltion test to extract the sane percentage of the trace, and its first P"""P^l 
Ixifextracted a nuch snaller percentage of the '^"-'^f ^0;;^^ s^^S^r 
knowledge test has even greater factorial complexity and has a 
-g.-neral factor" (18* as opro..ed to 34% extracted on the first principal axis). 
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ucort- for the lir:t of tr-,X)Ut questions in Task 2, by th«r following directions 
Gcores of Tasks 2 and I*, and ty the revisions and changes recoooended in- Task 3. 

■ae scorc-r. on Task 3 are even wore factorially coirplex since they .display 
substantial loadings on Factors 2, 3, 4, 5 and 6; ha^e>er. Factor 4 is nost 
-strongly associated with Task 2, since the P-F, F, SC, critique of tryout rsethod. 
and hanges rec-onaended in tryoat all show strong loadings cn Factor 4. Sim- 
ilarly, Task 4 has loadings on several' factors, but it is primarily associated 
with Factor 5. Factor 6, a nuch weaker factor, accounting for only five percent 
of tttv trace, is associated with several specific scores on different tasks, 
oDst notablyr the quality of instructions provided in the guidelines of Task 1, 
the handling of suggested revisions based on field test results of Task 3, and 
the- handling of visual elements and the quality of script (negative loading!) 
m TasX 4- Ihe scope scores of Tasks 1 and 4 also had smaller but significant 
loadings on the sixth factor. Kote also that all four of the scope scores 
display loadings of .25 or more on two , and sometime's three or four, factors; 
which tends to confira the character of this type of; score as the ability to 
apply a broad range or "scope" of knowledge or skill. ' 

To suzaarize, the factor analysis indicates that there are six factors 
associated with the simulation test- In terms of size of loadings. Task 1 is 
associated primarily with one factor (Mr 1) and to a lesser extent Task 4 is 
primarily associated with one factor (Hr 5) . However the scores in Tasks 2 
and 3 load on several different factors. We conclude that the simulation 
represents a cor^ilex of several different performance dimensions. 

Turning to the means in Table S4 note that the decimal part of the pass- 
fail score mearis is equal to the percent passing, thus only 21, 50, 24, and 30 
percent of the subjects "passed" Tasks 1, 2, 3, and 4 respectively. In retro- 
spect, the pass-fail standards appear to be too strict. The fraction means are 
6.53, 8.29, 7.29 and 7.36 on the 3-9 point scale. And all scope means are above 
3 on the 1-5 point scale. All eleven "specific" items and the following direc- 
tions and fractions items were all scored 0 to 9. ' Subjects did quite well on 

*The very high loadin gs for following directions in Tasks 2 and 3 are P^°^ably 
spurious since they can be attributed to extremely low scores made on both tasks 
by one or two persons. 
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Table B6 



Aaans for Three R&D Experience Groups, F-Ratlos and P-Levels For 
Tests of Differences on Sirnilation Test Scores 



Task 



Score 



R&O Experience 



no 
Exp 



ftodest 
Exp 



High 

EXD 



Group :ieans 



F-Ratic 



P-Level 



1 
1 
1 
1 
1 
1 
1 

1_ 

2 

2 

2 

2 

2 

2_ 

3 

3 

3 

3 

3 



Pass Fail (Guidelines) 

Fraction 

Scope 

Follow Oirectioris 
Consistency 

Specification of Objective 
Guideline Instructions 
Guideline Writing Quality 



1.15 
5:38 
2.69 
7-00 
4.31 
4.23 
7-54 
6.69 



1-29 

6- 76 
3-43 

7- 86 

5- 67 

6- 90 

8- 24 

7- 00 



1-19 

6- 84 
3-56 
8-22 
5-38 
5-6G 
8-19 

7- 34 



0- 51 
3-55 
2-71 
2-3? 

1- 89 

2- 96 
0-77 

3- 06 



.03 
.07 
-10 

fis 

.06 
fJS 
-05 



Pass Fail (Tryout) 
Fraction 
Scope 

Specification of Test Sites & Subjects 
Following Task 2 Directions 
List of Tryout Questions 



1- 08 
7-54 

2- 54 
5.38 
8.69 
8.46 



1-76 

8- 57 
3-86 

7- 81 

9- 00 

8- 56 



1.50 

8- 41 
3-41 

7- 12 

9- 00 

8- 84 



9-32 
5.50 
11-12 
7.45 



1.37 
1-20 
3.11 
6-24 

1-23 
2.40 
1-80 



.001 
.01 
-001 
-002 

NS 

»S 

JJS 
.05 
-004 

HS 

Its 

.10 

US 



3_ 

4 

4 

4 

4 

4 

4 



Pass Fail (Revisions) 
Fraction 
Scope 

Following Task 3 Directions 
Revisions Recomnended 
Critique of Tryout Method 
Changes Recbnmended 



1.08 
6-31 
2.62 
8.38 
7.69 
5.00 
7.46 



1.29 

7- 38 
3-52 
9-00 

8- 62 

6- 43 

7- 90 



1-28 

7- 62 
3-69 
9-00 

8- 44 
6-50 
8-62 



Pass Fail (Scri pt) 
Fraction 
Scope 

Following Task 4 Directions 
Visual Elanents.AV Script 
Quality of Script Outline 



1.08 
6.54 
2.54 
8.38 
5.92 
6-54 



1.38 
7-38 
3.14 
8.48 

6- 57 

7- 57 



1-34 

7- 69 
3-34 

8- 50 
7.06 
7-62 



2.03 
1.88 
2.17 
0.02 
1-34 
1-27 



NS 
HS 
NS 
NS 
HS 

m 



rot 

Tot 
Tot 
Tot 

tTot 



Pass Fail 

Fraction 

Scope 

Following Directions 
Consistency with Simulation Infor- 
mation* 



4.38 
25.77 
10.38 
32.46 
23.92 



5.71 
30.10 
13.95 
34.33 
29-67 



5-31 
30-56 
14.00 
34-72 
26-56 



5-38 
5.90 
9-17 
3.17 
7.03 



-007 

-005 

-001 

.05 

.002 



*The Consistency Tptal Score is the sum of the folla^/ing four task scores: 
Consistency in Task 1, Specification of Test Sites and Subjects in Task 2, 
Revisions Reconmended in Task 3, and Quality of Script Outline in Task 4. 
These four task scores best reflect the test subject's ability to cope with 
and respond consistently with the information (not directions) provided- 
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foliowing dirvctior>a with neans l.^ to 8.93. -Specific" scores with rela- 
tivelr hj<gn r>eaiisC8.t) or higher) aro: qui<3elinc instructions of Task 1, 
li::;t of tr^'ciat quei;tioi.ii of Task 2, and revisions recorsaended and <Aanges 
rocorstendcd both in Task 3. Relatively poorer perforriance is noted in the 
cas'i of cor^istency (of product guideline) with inforo ation provided and soec- 
xfication of instructional objectives , both in Task 1, critique of tryout nethod 
in Task 3, and handling of visual elenents in the script outline of Task 4. 

Kfiat about differences anong subjects in terzis of ^eir RSD experience? 
Turning to Table 6,; we see that scores on the four tasks varied in their dis- 
criaination of the three SSD experience groups. We note fijrst that the no RSD 
.:.xixjrience group h^s consistently lower oeans than either of the two eaojerienced 
groups on all 27 scores and on all the total scores. The laeans for the modest 
experience and the high experience groups are fairly close to each other. Ihe 
differences between groups on the fraction and scope scores in TaJks 1, 2, 
j:..J i a e all sicniZicant or near significant- In addition, 'the differences 
for guideline writing quality (Task 1) and specification of tes t sites and subjec ts 
(Task 2) are both significant. 3y contrast, none of the group differences for 
scores on Task 4 are significant. (Which may well suggest, that Task 4 could be 
easily oniitted fron the battery, except possibly in cases where specific training 
in audiovisual production is part of the. program of instruction.) - 

While not all of the individual item scores show significant differences, 
all of the total scores do. Perhaps most notable is the total for scope which 
has a P-level of less than .001. There are no appreciable differences between 
the modest and the high experience groups 'on the means for totals, but both 
of the experienced groups have means well above the no experience group on all 
five totals: pass-fail, fraction, scope, follgwing directions and consistency 
with simulation information. 

5. Relationships Between Self -Ratings, Knowledg e, and Performance 

Table 7 reports the intercorrelations among the total scales for the self- 
ratings, the knowledge test, and the simulation test. As we have noted in 
previous sections, all the total scores within the same instrument are signifi- 
cantly intercorrelated. However, our interest in Table 7 is in the correlation 
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betvec-n scores on the three different instnoMnts. Ihe results can be smii- 
narized briefly as follows: (1) all of the simulation and the knowledge total 
scoVes are significantly correlated (ranging fron .31 to .46), (2) none of 
tfie self-rating scales are significantly correlated with the simulation test, 
(3) sone of the knowledge test and the self-rating scales are significantly 
correlated. Knowledge scope c»rreiates .26 with the evaluation self-ratings, 
.41 with the product developmrat self-ratings, and . 31 with the technical 
writing self-ratings, the knowledge fraction score and the knowledge pass- 
fail score also diiTplay*very West correlations with these three self-ratings, 
with three of the six correlations reaching values significant at the .05 
level. It is significant that the context of the knowledge test iteas deals 
cost directly with product development, product evaluation and, to a lesser 
extent, technical writing, but does not deal with research, dissemnation or 
asD management. Ihe pattern of significant correlations between the knowledge 
test totals and three of the self-rating scales (but not with the other three 
self-rating scales) is as would be antrcipated. 

Hence the data in Table 7 indicate that the knowledge test scores are 
significantly related to simulation scores and to self-ratings based oncom- 
parable content areas. The unanticipated result in Table 7 is the fact that 
none of the self-rating total scores is significantly related to any of the 
simulation scores. Given the clear ability of the self-ratings to discriminate 
among groups with known differences in R&D experience, and given the' fact that 
(despite the relatively high intercorrelations among the six self-rating scales) 
the self-rating totals display a pattern of correlations whose magnitude and 
significance is consistent with the content of the knowledge test, we can only 
conclude that ^he self-rating scales are valid but that the three types of in- 
struments represent increasing degrees of competence specificity, with the self- 
ratings the most general, the knowledge test intermediate, and the simulation 
4:est the most specific. Some evidence supporting this explanation is 'founO* 
when we examine the relations between Biographic Information and the total 
scores for the three types of instruments. The self-rating scaleg are . 
significantly related to ir.embership in ^professional associations, authorsh i£ 
of journal articles , numbers of tests , RSD technical reports, and other 
publications . However, only the knowledge test scope score shows significant 
correlations with any of the ten R6D productivity measures; and in this case 
there were only t^o barely significant correlations, one with number of 
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evaluation rGpc»rts and the other with ms!!2>er of "other" products, lione of 
the sicsulaticn totals display significant correlations with any of the 5^ 
l#r<>di:ctivity rK.-*ii;ures. 

Taming to the Biographic Inforxsation data o n nmbers of courses taken # 
k:e found a 'irilar pattern, ihe largest nuzsber of and strongest correlations 
are found between the self-rating scales and courses taken/ with substantially 
fewer significant correlations between nu2±>er of courses taken and the knowledge 
test or the sinulation test. As we jaight expect, there are nodest to sub- 
stantial correlations (.41 to .63) between both the Research & Statistics 
and the Evaluation rating totals and the nur4>ers of courses taken in research 
design, in tests and xneasures^ and in statistics, as well as "other" (than 
educational) psychology courses. The Instructional Product Developiaent mating 
scale xs only correlated w^th nurber of tests and aeasurement coiirses (-27) 
and nuniber of statistics courses {.26); and the Dissemination Scale is only 
related to number .of research design courses (-29); however, the Technical 
Writing Scale is significantly related to number of COEtaunication Theory courses 
(.26), number of Test and Keasurenent courses (-33), nurtoer of other Psychology 
courses (.25) and nuinber of other Social Science «>urses (.26). The Hanagenent 
Scale mirrors the significant patterns found for the Bfisearch and Statistics 
and for the Evaluation scales; correlating with Research Design courses (.38), 
Tests and Measurement courses (-42), H-Jtistics courses (-31), and ofcier 
Psychology courses (.41). ' 

* In the case of the Knowledge Test, all three totals (F, P-F, SC) correlate 
(.27 to .30) with number of courses in Instructional Technology w The only other 
significant correlations are: -243 for knowledge P-F and number of Creative 
Writing courses, and .26 for knowledge scope and number of courses in Philo- 
sophy and Humanities. 

^ r 

There are only four barely significant correlations between courses and 
the simulation totals, namely: Simulation fraction score with number of other 
Social Science courses (.25); simulation P-F with Journalism courses (-.27) 
and simulation directions with Philosophy and Humanities courses (.26)- 
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*t-*-r*> isr*. oTi av€2ra9e of 5 significaiit correiaticns between nunSber of 
1 ^:iri,^r^ .aii^ aji'/ or^- of the ceif-ratirg scales, less than two significant 
'-'>xr*-l«jtai^ii*. i--?tw*:reri couxx^es ar*d on*^ of the knowledge test scores, and 
thai^ vT*e £*i^ificant oorrciation per sissulation score, toong the ten 
J5t-^x*urei, of KfcX> j^roducti vity , we fiiid an average of over four significant 
ci^rrelation^^ y*sx :;eif-rating scale, less than one significant correlaticn 
-1^. r knowl«id^c total, and no significant correlations with the ^ insulation 

s 

totals. 
ft, Suraary 

Oiven the sbove field test results, we conclude that the Educational* 
^'xoduct i)evel<;?pmerit Battery provides three different kinds of valid infonsation 
on DDfifE cor^tencies. Thr. self-rating instrument provides quit ^ broad but 
r*?latively superficial information on perceived cor^tent^ in six areas: 
he-^earcfi and Statistics, Evaluation, Instructional Product Developiaent, Tech« 
nxcal Sfriting, Dissemination, and R&D Kanageiaent. Although inoderately to 
highly intercorrelated, thi$ six scales provide significant and laeaningful 
t differentiations arong subjects with no Jl&D work e^sperience, subjects with 

nodest ejq^errence, and subjects with auch e^^rience. ftSojreover the self-rating 
scales display significrant and laeaningful correlations with nuinbers and types 
of courses taken and with nu2i>ers and types of R&D products authored or pro- 
duced. The three self-rating scales with content nost "sisiilar to the content 
of the knowledge test {developraent, evaluation, and technical writings) are 
sxgnxficantiy correlated witli the kriowledge test (Researdi S Statistics, 
A/is^enination, ^MD *$anageitent) . There are no significant correlations between 
tl^e self—rating scales, and the siinulation scale totals. This Usance of 
i»ianif leant correlations was not anticipated, but may be attributable to 
t-ic simulation test covering a far ctore specific area of conpetence concent 
than the self-rating instrurient. 

» 

The knowledge -test covers a much narrower range of content than the self- 
rating scales, yet it is broader in scope than tSie simulation. While only a 
.^est nuEber of the scores based on an individual 'question significantly discrin- 
xnate aisong the tnree groups with different levels of RSD experience,diffiTences 
on all three of the total scores are highly significant. The knowledge test does 
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Tj'^t da-icriEir-ate i^tveexj these vith hi^h and those vith aofiest StSD woi3: ex- 
i-"jrier.ce c-ut a.t dees discr ininate between these with experience and t3icE • 
iAc havv i-cr-e. r^-owlirdgc- Test sec-res are si-gnificantiy bat very modestly 
related to tiie -self-rating licales «ith corresponding general cssntent {Instruc- 
ticnal Product I«velcpaenl, Evaluation, Technical Kriting) , and to nuzber of 
ccarses taken m ir^tructicnal technology. The Kncwledge Scope score is also 
siqnxfxcantly related to ntadaer of evaluation reports authored and ntiiiber of 
-other" products produced. 'She Kncwledge Test (which vras designed to include 
content roz:^iy paralleling that of the Simulation Test) is^ significantly 
Lut only nodes tly correlated (-25 to -46) vith every one of the Sismlaticn 
Tf-t total scores - 

All the Sxrnilatxon Test total soures (in addition to their significant 
correiatxon vith the kijovledge test totals) discriainate significantly between 
those vith no &£D work experience and those with PSD work experience. However, 
as xn the case of the Knowledge Test, Ihe Simulation Test was designed to 
appraxse "entry i*»vei- cor?)etence and not higher levels- Ifeither instrument 
discrxmxnates between subjects with relatively modest SSD essperience (average 
of 1.4 years) and those with much greater RSD experience (average 10.1 years). 
Perhaps because the Sisailation Tjsst deals with a much narrower range of cosa- 
petencies, it displays no significant correlations with any of the six self- 
rating scales, or with any of the ten measures of R&D productivity. 

Each of the three 'instruments tends to provide different kinds and 
amounts of information- The lack of strong correlations among the three 
types of instruments indicated that one instrument can not be substituted for 
another. *'jtoreover, aside fron the moderate to strong xntercorrelations among 
the six self-rating scales, the correlations between knowledge questions or 

*Using HO RSD Work Ex perience (=0) vs Kod^t or High psD Work Experience (=1) 
as a to/ criterion, and employing the total scores as Predxccors we obtaxned 
?L following Dultiple correlations: .46 for the six Self-Ratxng scales; .42 
Tr lleZrTo SUedge Test Scales; .50 for the five Simulation Scales; .3 for 
SeU-^Itings and Kno-iedge Test, .53 for Kna.ledge Test and Sxmulatxon Test^l 
for Setf-.^tings and Simulation, an.^ .62 for all three xnstruments. All ofO^ese 
nuL!ple ^rreLtions are significantly different from zero 
S^ar Se Slf-Ratings plus the Simulation produces almost as hxgh ^ ™ljxple 
that the Sel. P3Cx g p ^ coinbined. From this point of vxew the Know- 

S re ?ertVrovtL''vrr; Uttle additional predictive i-^r^tion beyond thaj 
supplied in the Self-Eatings a.nd the Simulation. But note carefully that -these 
results are relevant to this one crxterxon- 
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ar^-latjrj, r.^.r^u witLii. it -Ar uiDC-ntr are not ctror.g; hccce there is :;ubstaritial 
.'--■-sf jcaty th*-^ tt_-Tj hr*fwU-dqe qiK^tionr* £ind th^* fcur iJimilatiorj task^j. 

ires tri- .-t^-ii^i,2r.t of ^ore:* attained, all throe typos of instnuaent^ 
^ar^riEkiT-ate it :^tutisticaliy r*i^ificar*t leveiis betveei^ those with no E&ID 
w rk exf^ri<fr.ce oi^^ those with sccoe RSD work coper ier.ce. Hcwever, highly 
^-xt*c-riejjced persons do r.ot nec<^-skrily do better than U-oz ejsperienced per- 
i^cf:^ en either the hnc5wledge or the sisauLation tests. Cniy the self -rating 
i;c»3lei', v-.pecialiy Dissemination and Kanageiaent, separate the hi9hly 
•-«>««-ri<-T.c<-d from those with nofe 3X>dest I^B work experience. 

In cea^- of scale differences, those with no escperience tend to rate 
tt>.^elvei5 about one perfonaance level lower than the highly eD5)erienced 
|-can perfona if given ejx>ugh general st5>ervision or general instructions* 
v," -cai. perfona quite satisfactorily without supervision or job aids") about 
t.-n i^roentage points lower on the knowledge test (781 vs 87% of possible score 
on traction score and 63% vs 74% on scope score), and thirteen percentage 
^<»ints lower on the siimilation test fraction score totals (72% vs 85% of poss- 
ible score) and eighteen percentage points lower on the sisaulation test scope 
score totals {52% vs 7C)%). 

In retrospect, we wish that we had include a group of students who 
had just entered the DB&E progran and were without significant relevant 
training since we expect that the differences observed for such a group would . 
be even larger. Please recall that even our i:o R£D Kork Experience group 
had taken courses in Research Design, Tests and Measurements Statistics, 
instructional Technology and Evaluation. Hence their self-appraisal, knowledge, 
end oerf^'-^ance skills would be ejqpected to be substantially above those 
with neither relevant academe training nor relevant R&D work experience. 
However, our primary intent was to "calibrate" the battery against advanced 
students and experienced developers, since it is unlikely that the battery 
would ever be used in practice with entering students. 

Given the results reported, we reconzDcnd that simulation Task 4 be 
deleted since it is the only task which showed no significant discrimination 
between the R&D experience groups. This should reduce the simulation to 
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f^xitD&iK'ly »hr»re hcuTi. duratir-ij. Secor^d. we r«xiamend deleting at least 
cr ^hr»- fcnowl-e^go q:uc-:^t :orjr or altttmatively extending the one hour tis* 
Ijni^ -OTj tfce knowU-^cjC' to;;t- Th*- one hour time. limit for ten essay questiotiS 
ir; ^jrt-ascr^l*-'. It tia^", to place too nuch er&hasis on "speed" and too 
little "power." Finally, we arci& the user to consider lowering the pass- 
fail criteria -Ijiai are indicated for the knowledge qtsestions and the siimia- 
taon tasks. Although these crit*.ria seeaed to be reasonable at the tiiae they 
ar-- unreal asticali/ high in terins of the performance of esa^rienced srtojects. 



*Tne pass-fail cri^ria which were applied in this field test required that 
tie subject's response include all or virtually all of a nunber of eieaents. 
In :iinst cases, the subject "failed" if even one element was missing. 
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TO: Assessment Subjects 

FROM: Laird Blackweli and Paul Hood 

DATE: Hay 2. 1975 

SL'SJECT: The Instnictional Product Development and Evaluation 
Ccsjsetence Assessaent Battery 



For the past year we have been developing a model assessment battery 
designed to assess both knowledge and skills requisite or useful in tasks 
that are frequently encountered in instructional product developn:ent. In 
several pilot tests, we have discovered that self-ratings, written ^/f^^V^^-^ 
tions, and job sanple simulations each have a degree of discnininant validity, 
but that each also contributes information not provided by the other. 

Me are nov; in the process of validating a revised version of the Compe- 
tence Assessnient Battery. Although the Battery focuses primarily on entry- 
level" (H A degree level) competencies, in ordar to develop validation data, 
wo have enlisted the cooperation of test subjects representing a very jroad 
range of coirjpetencies, including students in graduate programs and highly- 
experienced RSD personnel located in RSD agencies throughout the country. 

normally, a student or on-the-job trainee would take the Battery after 
completing a related series of training modules which have been developed 
by the Far West Consortium for DD&E training. The information derived 
should provide the student and the instructor (or work supervisor in the 
case of an on-the-job training progran) with some useful guidelines as to 
the degree of competence in several areas related to instructional product 
development. Please note that the current Battery makes no claim for compre- 
hensiveness. For instance, a number of important skills and knowledge areas, 
e.g., task analysis, specifying learning sequences, are not incl"dea «atner 
the Battery represents a model for developing test instrumentation that goes 
beyond the common written examination. 

Self- Rating 

You are asked to complete two self -rating instruments- One consists of 
61 statements of activities encountered in instructional research, development, 
dissemination and evaluation (RDD&E). This instrument has been designed to 
help us "profile" your own assessment of your "level of performance in a 
broad range of RDD&E activities. A second, much shorter instrument, ca Is for 
you to rate your "level of knowledge" on 11 items which are most directly 
related to the kno-//ledge and simulation tests of the Battery. (Please do 
not refer to your "level of performance" ratings while making these level 
of knowledge" ratings; we have found in pilot testing that the two ratings 
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are highly, but by no peans perfectly correlated.) Although the ol-Uea 
self-rating instrunent includes activities over a broad range of R,D,D, and 
£, it eT4}hasizes those involved in educational developn:ent and evaluation 
{sQ-^e of which are directly related to those activities encountered in the 
YjimAedge Test and/or siriulation exercises). There is no tice limit on the 
self- ratings- i^ost subjects will find that they require less than 1/2 hour 
to corplete. 

Knowledge Test 

J^ost of the questions in the Job Knowledge Test are designed to assess 
enabling knowledge for the sinjulation tasks. These questions should be use- 
ful in identifying gaps or weaknesses in your knowledge about educational 
deveiopn:ent which would likely hinder your performance of soise of the tasks 
a developer wjuld frequently be required to perform. This test has a tiine 
limit of 1 hour. 

Simulation 

In the series of four siimilation jexeriiLses,.yi)a ace^ked io^perform 

several tasks. All the necessary instructions and materials are included 
in written form. Time limits are suggested for each task; the entire series 
should take no longer than 3 1/2-4 hours. 

It can be seen that as the method of assessment approaches observation 
of on-the-job performance (i.e., as you go from self-ratings to knowledge 
test to simulation exercises), the scope of corr^jetencies assessed gets 
progressively narrower (see Figure 1 on the following page). If, as 
preliminary data analysis from field testing of the first form of these 
instruments suggests, each of these three instruments provides valid but 
largely independent information (i.e., each instrument discriminates Between 
students and professionals, but has only modest correlations with the other 
instruments), the Competence Battery may be more useful as an example and 
guide to further instrument development than as an assessment tool for 
instructors and supervisors over a wide range of educational R,D,D, and E. 
Instructors/supervisors may want to construct knowledge items and/or simula- 
tion exercises relevant to their particular area of interest or concern. 

In order to f5rovide us v/ith information on the validity and usefulness 
of this Battery, wa would appreciate your filling out the DD8E Student 
Biographic Data Form regarding your relevant employment, education, training, 
etc., and then completing the self-ratings of performance, the self-ratings 
of khov/ledge, the knov/ledge test, and the simulations in this order. The 
entire sequence should take no longer than 5-6 hours; you may want to 
break this into tv/o or three sessions. 

Please note that all products of your work and all forms you complete 
v/ill be treated as confidential information and v/ill be used only for test 
development purposes. The set of response forms you have received are 
marked with an identification number so that the various pieces can be kept 
as one file. This is the only identification used. A supply of paper is 
provided for your convenience in completing the products called for in the 
simulation task, (You may use your own supply of paper if you wish,) Please 
be sure to mark your identification number c-» the top of each page of your 
products, 
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Figure 1 
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With your Competence Battery materials, you have received a contract 
to complete (so we can send you your payment) and a return envelope 
addressed to us at Far West Laboratory. To expedite our receipt of your 
completed Battery, please enclose all materials (i.e., coinpleted biographic 
form, self-ratings, knowledge test, simulations. Task Evaluation forms, and 
signed contract) in the envelope and return it as soon as possible to us. 
Thank you for your cooperation. 



LB/PH/cb 
DD8E/CAB/4-75 
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R,D,D,&E ACTIVITIES 

Tmse self-ratinq items focus on activities related to educational 
P,D,D,^. For each iten, indicate how well you feel you can perfona 
the activity by writing the appropriate number from the following six- 
point scale on the line next to the item. (Please reinove this page so 
you can refer to the perfonnance levels listed below while you conplete 
the ratings.) 



Level of Performance 

1. I cannot perfoim this activity even with supervision 
or guidance. 

2. I can perform this activity if I have either detailed 
instructions or close supervision. 

3. I can perform this activity if I am given enough general 
supervision or general instructions. 

4. I can perform this activity guite satisfactorily without 
supervision or job aids. 

5. I can perform this activity guickly and efficiently and 
can do a top quality job. 

6. I consider myself an expert In this activity and c^n 
accomplish unusually difficult or completely novel 
;/ork. * 



DDgE/CAB/4-75 



f 



17u 



R,D,0,&£ Activities 

Fill in the appropriate 
nimher fron the scale on 
the previous page. 



1. Revise a test instrument on the basis of • 
data collected durinq a pilot test of ihe 
instrument- 

2. After product installation, interview users 
to detennine if adequate infonnation was pro- 
vided about how to install and use the product. 

3- Choose (or design) appropriate statistical 
techniques for data analysis. 

4. Use bibliographic resources such as Research 
in Education and Current Index to Journals in 
Education to locate information supporting the 
need to develop an educational program i)r 
product. 

5- Design a monitoring system that will provide • 
data on the status of the operating system • 
(such as actual vs. intended outcomes, unmet 
needs, problems, etc.). 

6. Supervise professional R&D personnel. 

7. Fran a general description of a product's 
purposes, specify appropriate terminal ob- 
jectives. 

8. Prepare a sample of an educational product 
v/hich can be presented to a specified target 
group in order to determine if the proposed 
product meets the group's needs. 

9* Review first draft materials to determine if 
they comply with specifications regarding 
objectives, sequencing, and content. 

10. Derive a set of questions which could be sent 
v/ith your product to reviev/ers in order to 
focus their review on critical aspects of 
your product. 

11. Use and interpret statistical regression 
techniques^. 
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12. Identify the types of questions which can 
and cannot be ans^^ered by different research 
designs. 

13. Design data collection procedures to maintain 
privacy or confidentiality in collecting, pro- 
cessing and storing infonnation. 

14. tonduct a search to determine if oroducts exist 
which could meet your need(s) for instructional 
riaterials. 

Confei with different types of users about their 
problems in the use of an educational, product. 

From a summary report of a tryout, specify the 
problenjs that need to be remedied by product 
revision. 

Write and submit proposals to obtain funding 
and to'negotiate>ith funding agencies. 

18. Detennine if new test instruinents need to be 
developed by reviewing how well available tests 
fit the evaluation objectives. 

19. Identify and articulate the problem in a re- 
search study. 

20. Determine financial resources necessary to 
conduct a program or project and use accounting 
procedures to operate within a program or pro- 
ject budget. 

21. Given the summary report of an initial tryout, 
suggest new methods to use in the second tryout 
to improve tryout validity. 

22. Given a problem statement, information on the 
history of the problem, objectives and possible 
solutions, write a 2D-page proposal for solving 
the problem, including a rationale for the ap- 
proach and a development schedule. 

23. Formulate testable hypotheses in a research 
study. 

24. Prepare a coding scheme which will be used by 
a group of coders in preparing field test data 
for computer analysis. 



15. 
16. 

17. 
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25. Kake arrangeiseiite to secure cooyrfghts and 

copyright clearances Khere needed. 

26. Given an educational product Khich you are 
developing, prepare tests and rseasures to 
assess stadent perfprsance. 

27- Identify factors which can jeopai^ize inter- 
nal and external validity. 

23. Convert a chapter In an instnictional text 

into a branching progranESd instruction ^ 
sequence. 

. 29. Detennine the theroughness of distribution 

Khich ocairred in dissemination of an educa- 
tional produqt. 

M 

30. in planning the, evaluation of a new training 
program, determine the types, of subjects which 
should be used. 

31. Mrite a detailed product description including 
all the necessary elements. 

32. Prepare a product design which is consistent 
with the stated overall purpose and use of 

^ the product. 

• 33, Enjploy sampling theory and techniques to 
■^vl — develop a stratified or clastered sampling 

• plan. 

34. With production personnel, determine the se- 
quence of activities for production of instruc- 
tional materials. 

35. Review a draft field test report to detenalne if 
: - it is correct, coniplete and ready for. release. 

36. Deliypr an oral presentation to a group of more 
than 20 professional educators to disseminate 

infonriation about a project or product. 

37. Plan Interviews with potential ysers for tne pur- 
— ' pose of determining a market for your product. 

38. Locate existing methods or strategies which can 
be used in potential product development. 
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39. Krite scripts for instractional fil?^, video- - 

tapes, or sound filrstrips. 

40. Categorize instructional objectives in teras 
of a taxcnony (e.g., Blooni's Taxqnony of the 
cognitive doisain or Gagne's conditions of 
learning). 

41. Collect inforaation on installation costs for 
your instnjctional product. 

42. Detenaine an experimental design and statis- 

; tical techniques vhose underlying theoretical 

assumptions are consistent. 

43. * Identify the crucial characteristics of a 

" target group Khich may influence the dissemi- 

nation effort. 

44. Provided vnUi product specifi cations r the test 
product, field test reports, and expert reviews, 

nake recomendations regarding the nature and 
extent of revisions required. 

45. Prepare a memorandua which sunniarizes the ac- 
tions taken and problems encountered in a 

\ !neeting with your development team. 

V 46. Mrite exercises Khich the learner should do 

in order to master concepts or principles in 

an instructional unit. 

47. Evaluate alternative media and instructional 
" methods to match them with instructional ob- 
jectives and learner characteristics. 

48. Evaluate the feasibility and risks associated 
with alternative approaches for redesigning 

a product that has failed to meet specifications, 
/ 

49. Design data collection forms for a marketing 
study of an educational product. 

50- Prepare ^ nummary report of the test of your - 
Instructional product. 

51. In preparing a proposal for a development pro- 

ject, sfearch for and select references which 

would support the. rational and technical ap- 
proach of the project. ly^ 
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52. Deteraine revisicns that can be rade witJbin 
specified tire constraints. 

53. P-eview the relevant research literature for 
evidence to support the selection of an in- 
structional strategy. 

54. Deteraine the effectiveness of using a denon- 
straticn to diss^inate infonnation to your 
target audience. 

55. Vse n^nagenent and planning systems such as 
PERT {Progra!3 Evaluation and P^view Technique), 
PPBS (Program Planning Budgeting System), or 

Critical PatiL Analysis. ^ _ _ 

56. Evaluate at least three different narket re- . 
search techniques applicable to a specific 
developinent. 

57. Specify a set of procedures to be followed 

by evaluation personnel in adniinistering test 
instnaiaents. 

58. /inalyze job or task requireiaents to detersaine 
objectives for an instructional program or 
product. 

59. Detennine questions you would like to have 
answered by the data collected in a tryout of 
your product. 

60. Obtain information about strategies used to 
market educational products similar to yours. 

61. Analyze discrepancies between expected and actual 
test outcomes. 
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Job K53Dw1edge Questions 

Kost of these questions were designed to assess knoMledge that Kould be 
necessary or useful in perfonsing sc^ of the tasks of an educational developer. 
Use the space available to answer the questions briefly but completely , in 
aany cases a sinsple list will suffice, if you have any problejss or questions • 
ask the proctor or cooixiinator for help. 

Please allow 1 to 1 1/4 hKjrs to cccplete this task. 



I 
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lE-.dt iirp&rtant infonaation that should be included in a description 
of a product for a developer is missing froa the following capsule 
version of such a description? 

Ite "Victorian England" course will serve as introduction to 
art, literature, and social life in England during the Vic- 
torian period. The course will include selected readings frca 
texts on Victorian social life and the arts, but will emphasize 
the visual arts and literature of the period as instruction in 
those areas as well as sources of infonuation about social 
customs. "Victorian England," when produced and packaged, 
will include the folloiring: 1) a book of readings, for each 
student, including works by and about residents of Victorian 
England, and 2) a paperback book, for each student, of black- 
ai;*-white photographs of Victorian painting, sculpture, and 
archi tecture. 

'jkers will be required to read the required text, but user 
'activities will foois on analyzing and discussing reproductions 
of Victorian art and several of the best-known novels of Dickens. 
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?- Cn the basis of tl:e following descripticn, specify appropriate terainal 
objectives for this product. 

The -Plant Life of Kew Zealand** files will include still photo- 
graphs as well as inovie sequences of Kew Zealand's c^st cocaaon 
plants both wild and dossestic. The filsn will be divided into 
several segnsnts, each covering a different fanjily of plants. 
Each segsent will be followed by a short quiz or plant identi- 
fication and associated knowledge- The fila is intended to 
faniliarize the high school biology or eralogy staxdent with 
JJew Zealand's OCTson plants and their faniilies and to tsach the 
sbident what plants are edible and what plants are poisonous- 
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3. An mstmctional fllsj cn the behavior and attitodes a woman might adopt 
during pregnancy to help her prepare for child-birth has been developed 
fc/ a connunity health ser/ice to be used in conjunction with reading, 
discission, and practice in a "store-frent" weekly course for pregrant 
wotsen. in order to try out this filsa on as cany people witn as rew 
adainistrative problsas as possible, the developers plan to show it to 
several hygiene classes in a nearby high school and then test for know- 
ledge as well as interview for attitudes toirard the nla. 

On the basis of this description, discuss briefly at least four aspects 
of this proposed try-cat that could affect the relevance and importance 
of the information it would yield. 



131 



4. 



4 



Mhy a tryout useful before a field test? Khy not onit the try-(r:t and 
proceed to the field test as soon as possible? 



5. Briefly crapare the advantages and disadvantages of using photographs of 

real subjects vs. drawings ¥i cartoons for a film sequence. 

or 



(a) 
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6- A prograsrced text has been developed to teach high-sdaool st^idents the 
rudiments of statistical inference. Indicate ttree questions a developer 
rjight want to answer in a tryout about user satisfaction. Suggest two 
y;etljods for collecting information on each of these questions. 



7. What are the most imporUnt kinds of infonnatlon a script writer needs to 
know in order to produce an instructional film? 
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8- Your RiD agency has been preparing a science unit for grades 2-4- The 
unit includes a cnovie, student KOrkbook, and teacher guidebook, for 
eacf3 of six subjects presented. Listed below are questions which de- 
velopers of the science unit would like to have answered by reviewers. 
For each question, suggest at least one type of person who could review 
the product help answer the question (e.g., Hedfa Specialist). 

1. Are the goals of the product valid and reasonable? 

2. Is the content provided correct? 

3. Is the media used appropriate for the target audience? 

4. is the organization and sequencing of the content effective? 



5. Is the AV equipment required easily obtained by people who 
will be using the product? 



6. is the language level appropriate for the target audience? 



(a) 
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A large oil coespany has asked your Ril) agfSicy to develop a method of getting 
infomation about the uses and processing of petroleun and the present 
research taking place in the industry to upper elexentary schools, A 
decision was made to develop a self-contained presentation in the ^'onii of 
a losm, 45 ainute colored fila. The film, shgt on location, explains hOi* 
petroleusa is refined, why new methods of refinesnent are needed and what 
is being done to find new oethods. In addition to the film, a short book- 
let for each student has been developed which shows, in a cartoon sequence, 
the many uses petroleum has in our society. 

Although you have been working with the content expert provided by the 
conpany, no one else outside your agency has seen your product- Be.lcfw are 
three people who have agreed to review your product. For each reviewer, 
list at least three questions you would ask then about your product. 

1. A 5th grade teacher 

'2. A sixth grade student 

3- The Oil Company Representative who initially asked you to 
produce the pri.duct. 
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If the subjects you have selected for the tryout of your product have 
difficulty in using the product effectively, what factors other than 
the product itself could have caused the difficulty. Specify three 
causes and, for each, suggest a resnedy. 



Outline the sequence of major steps (at least five but not more than ten) 
that are niost important in the production of a sound filmstrip between 
(a) the definition of the instructional objectives and (fa) the collection 
of the first valSdation (pi^ot test) data. 
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fntrinsic motivation (when the learner finds performance of the task itself 
rewarding) and extrinsic motivation (when a person's reasons for doing a 
task lie outside the task) can both be powerful forces influencing learn- 
ing. Discuss how tests are" used in Instructional laaterials to enhance 
a) intrinsic motivation and b) extrinsic motivation. 
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Briefly define and describe the relationship to (or place in) educational 
develcpoent of each tem listed below. 



13. Field test 



14. Enabling objectives 



15. C^petence donain 



/ 16. content validity 
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17. fomsative evaluation 



18. Contirfgency ioanagesent planning 



19- Prototype 



20. Constraints 
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PU>se T««l tills before attf^lnf ttfk 1. 
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AN EXERCISE IX EOUCXTIONAL DEVELOPMOfT 



D»petence-based educational programs need soae aeans for assessing 
students* coapetency. Often .the assessment dQiends priaarily on tests of 
knowledge* Honeyer^ It is desirable to assess perfomnce as iiell. 16 soae 
extent this can be done evaluating the rarlous products a student produces 
as the outcoae of special projects or as part of their internship work. One 
problea in basing assessment solely on such products is that there are aany 
uncontrolled factors; for instance^ hoir iKich help was given or how such tiae 
was available. To overcone this lack of standardization, a siiHilation test 
has been developed^ 

Because this kind of simulation is an entirely new experience for »any 
persons, we urge you -to read this introduction. It my help to answer soae of 
your questions and it should enable you to understand how the siaulation is 
organized and iiriiat you will be expected to do. 

This sirailation will require ycu to perfora oany things that an educational 
product developer wwld need to accomplish. The tasks have been chosen to 
provide a "sanple" of a nunber of specific developnent coapetencies that are 
frequently encountered in product developnent work. The tasks included in this 
simulation exercise have been designed to focus on essentials and bypass a 
lot of the less relevant detail. This has been achieved in several ways. 

First, a relatively sinple, and yet not trivial development task is used. - 
You will find that the subject matter is sufficiently simple that no speciVl 
<onjent expertise is' required. Second, the sequence of tasks is organized sto 
that tte infoRsatfon you receive in earlier tasks can be used in performing later 
tasks. This saves you from having to do a lot of reading about entirely unrelated 
tasks, but more imporUntly, you will find that the information available 
(although never enough to answer all your questions) wiil accumulate until, 
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in the later tasks » you ate confronted Mtitti a fairly realistic deyelopoDent 
situation which has a meaningful and relevant '"history-'' Third, varicMiS kinds 
of information are provided (e,g., "iDodels," instructions, lists of questions, 
etc.) vhich help to define what jfotf are to do and which supply a continuity of 
sorts In tenns of what "happened** between each task- 

Soaething we've keptinssind in constructing this exercise is that develop- 
ment is a teaa effort, with usually som d^^^gree of specialization of work. 
Alraost always you can count on being able to interact with others for advice or 
assistance. The inforaati^i and instructions you will be given in this sinula- 
tion atteicpt to serve as the interaction you would ordinarily have with your ^, 
supervisor and other team jaembers. Furthenaore, you will find ,that several 
"team decisions" will be made during the simulation. These decisions in effect 
say, ^'Your work has been considered and the teas has decided that this is idiat 
we should do next..." 

You will find that nearly all instructions and "coramuni cations" have been 
preplanned and are in written form. tt)viously, we are unable to anticipate every 
problem you laay encounter. You should try to work on the basis of the written 
information and instructions given you. While your questions and cooinents are 
encouraged and even requested at certain points in this siniulation exercise, we 
would like you to please try to confine your questions and coramenls to writing. 
But, if you have a significant problem that cannot be handled in writing, ask 
the test administrator for help. 

'J*ote that this is not a strictly speeded test, but there are time limits. 
At nearly every stage, youinay feel that the time allotted is insufficient. 
Do the best that you can. Hake sure you finish at least the essentials. The 
time limits have been set so that most beginning developers can complete at 
least a "rough draft" if they have been efficient in their use of tiwe. Read 
your instructions carefully. Make sure you understand what you are to do* You 
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my want to Jot dom tiM soany minutes you can spend on each part of your plan 
of irork» and then keep track of the tisae. Quality generally counts more than 
quantity^ but if you fail to allaj yourself the tine needed to produce the 
tangible docui3ent(s} required at eacii stage of this siiaulation, you my find 
It difficult to coinplete the task' within the tiae liisit. If you finish early, 
review your work and juake any revisions or additions which you think will im- 
prove it. Scoring will be based on the quality of your work accowplished within 
the tine limit . 

Please note that your performance will be assessed on several dimensions: 
completeness, appropriateness to instructions and background information 
provided, originality of ideas, sensitivity to problems and issues, technical 
quality of products, and quality of writing. Each task required of you focuses 
on a related set of product development knowledges and skills. The tasks cover 
a broad range of competence levels, but are focused primarily on the kinds of 
tasks a beginning developer (at least a college graduate with one year of expe- 
rience or training in product development) would be expected to perform with 
sane, but not a great amount of supervisiwi. The total novice to product de- 
velopment may find the task difficult but not impossible. 

We hope you find this exercise a challenging and useful expr-ience. 

* * * 



You should now read "Your Role as a Member of the Development Team." 
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YOUR ROLE AS A iCWER OF THE rZVELOPiBJT TEAH 

Please icagine you have been esplcryed by an organization which has sub- 
contracted ifith the Far West Laboratory for Educational Research and Developr 
inent^to produce parent training materials for the Parent/Diild Toy-Lending 
Library • A deyelo{»Dent teaa at the" Far West Laboratory has been working to 
create the toys involved- in the Toy-Lending Library, Most of tiie toys have 
already been tried out wftt children* The toys ^re^^lher being^jrevised^c: _ 
are ready for production. Your organization is developing a package of 
inaterials to be used to train parents of three- and f cur-year-old children 
how. the toys and games can be used for learning episodes. 

The^ team youliave joined has been at work for a relatively short tiiae. 
They have produced an initial version (a prototype) of the materials needed 
for the first parent training session and have tried it out with a few test 
subjects. The results have been encouraging. 

You have been assigned to the parent- training session on the "Feely Bags" 
toy. In the course of your work you can ccunt on the following kinds of 
assistance: 

1. The teaan leader will supervise your work. However, because of other 
^pressing cocimitments at the njomenc, you can expect only very general 

instructions and guidance. 

2. The secretarial services pool will type your drafts of instructional 
fnaterials, memos, letters, reports, etc. You must be sure to write 
legibly and fo provide complete Instructions. 

3. Production Services will produce filmstrips from scripts; art 
work from ;etches and do printing and packaging work. 
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4. A research and evaluation team has laade arrangenients to conduct all 


of the project field testing.. JTou- otJI need to conunicate your 


initial (tryout) requirenjents to thea. 


In order to better acquaint you with the parent training sessions, your 


supervisor has referred to you tte folloMring section of the original proposal 


irfiich describes the general fonn of the materials for the parent training 


sessions. 




Fov each parent training session a segaraie seZf^contctined pGckage of 






nxzterials^ called a Resource Kit^ will he produced. SfZis Sesource Kit ti>ill 






contain all the materials needed for one session to instnict parents in the 






use oj oTi^ L>uij^ 




- 


The training provided by ihe materipCls in ihe kit should not take 
wore than two hours. The materials should not use tecSmical terms 
and should be written clearly (at no more than the 8th grade reading 
level). Any equipment (audio-in^sual or other) needed for the session 
should be the type aoaildble in most schools or commodity centers^ or 
easily Tinted. 






The materials which will be found in eacSi Resource Kit are: 






I. A Librarian's (course leader's) Manual to provide all in- 
foTfT^tion ti^cessaru 'f/rh a. X^/zdsT* nnjR trMiyt77i/] R^j^ti'LcTi 
Jhstructions to Librarians will be double-spaced^ numbered, 
and listed in order as they are to be used. Any section of 
the manual designed to be read aloud wilT^be triple spaced 
and boxed: 






2. A filmstrip presentation showing a parent and child using 
the toy. 






• 

d. A toy and set of instructions for each parent* 






The Librarian will read the Librarian's Manual before conducting the 
session and will refer to it during the session. The parents will be 
required to listen to an introduction of the toy by the Librarian^ ob- 
serve the filmstrip presentaticn^ and take part in a role-playing session 
led by the Librarian. 






After reading the Librarian's Manual^ the Librarian should he able to 
plan the session, introduce liie toy and its purpose, organic fte rote- 
playing eession, review the Responsive Frogram principlee, and answer 
questions asTced by the parents. 
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After the training session^ the pczrents shculd be able to dsK^crihe 
in their oim words: 

Z- 5Sie "gvrgoBe of the specific toy. 

5- The gcunes t02ich can be played with the toy. 

3. At Zcust one principle of rhe Responsive Program^ 

^. An ezc3wle of one of Hie Responsive Progrcsn^s principles 
fcrjond in iSne axjidio-^visual presentation* 



The following section will provide background information you may need 
about the Parent/Child Toy-Lending Library- 
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THE PARENT/CHILD T0Y-LEHDIH6 LIBRART 

The Parent/Child Toy-Lending Library developed by the Far West laboratory 
for Educational Research and Development is designed to help parents participate 
in their children's learning. The program shows parents of three- and four- 

* 

year-old children hew toys and games can be used for learning episodes and allows 
them to borrow toys as one borrows a book from a library. In essence, the 
developers intend to create a system to change parent's attitudes about their 
children's learning potential and about their own competency. In addition to 
developing certain toys, the developers envision a parent training program as 
an important part of the Toy-Lending Library and will offer an eight-vfeek course 
for parents which will meet once a week for about two hours. Without this parent 
training , the product degenerates into a package of toys, none of which is 
unique in itself. 

At each weekly meeting, a new toy will be introduced, the concept it at- 
tempts to teach will be explained,' and parents will become familiar with ways of 
using the toy. The new toys are then taken home for a week. When the course ends, 
parents will be encouraged to use the library as a permanent source of additional 
toys, games, books, and records. ' / 

The program's goal is to help parents promote the intellectual development 
of children in a W2ty that is likely to support the development of a healthy 
self-concept. It is designed to foster children's growth in concepts and skills 
such as color, shapes, problem-solving and verbal, coninuni cation. The general goals 
of the training sessions include: 
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' 1. Parents will feel that they are more competent in helping 
their children 'learn some important skills and concepts. 

2. Parents will feel that they can influence the decisions 
that affect the education of their children. 

3. Parents will feel that the child is capable of learning 
and can be successful . ' 

4. The child increases his competency as a result of the 
interactipn with tfie parents.' - . ' 

Far West Laboratory's Responsive Program 

The developers' goals for the Toy Library reflect the long-range 

objectives of the Far^West Laboratory's Responsive Program, an educational 

system being developed for children aqsd three to nine. These objectives 

are "to help children develop a healthy self-concept as it relates to 

learni.ig in the school and the home, and to develop their intellectual 
2 

ability." A diversity of educational experiences is sought ty the Re- 
sponsive Program, which rests on the assumptions that much learning takes 
place in the home, that formal education must be supplemented and that 
the educational program must be closely tied to, the child's culture and 
background. The responsive environment leaves a great deal of autonomy 
to the child: adult-initiated talk should take second place to child- 
initiated talk, and the child can choose not to participate in group 
activities (as long as he or she does hot disturb the group) In addition, 
the child's activities should not depend upon efxtrinsic rewards or punishments 
and they should help in developing a useful skill, concept or attitude, 

"^ A Guide to Securing and Installing the Parent/Child Toy- Lending Library. 
Far West Laboratory for Educational Research and Development, Berkel ey , 
California/1972, C5. ^ . 

2a Guide to Securing and Installing the Parent/Child ToyrL ending Library. 
Par West Laboratory for Educational Research and Development, Berkeley , 
California/1972, B3. 
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The Toy Library was conceptualized as a way to help parents create 

a responsive environment for learning in the home. Its principles are 

those of the Responsive Program as a whole: 

J. Free exploration : The child is free to explore any toy 
.and to change ^he rules of any game he or she may be playing 
^ \iiih the parent. . -^ " 

2. Self-pacing : The child is free to work at his chosen speed 

• arid to stop work when the game or toy is no longer of interest. 

3. Self-correction : The toys are so constructed that the child 
can Imnediately find out the results of explorations, either 
from the toy itself or from the parent. 

4* Discovery learning : The child is given time to discover 

things for him or herself—the parent helps the child to 

think through a problem rather than giving the correct 
answer. 

5. Self- reward : The. learning activities are satisfying to the 

child. The game is not played because the parent will reward 

or punish for not playing, the child learns because he wants to. 

The principal developer of th^e Responsive Program sees the Toy 

Library as a way of enhancing parents' Involvement with their children's 

education. One staff member states that the deyeloper: 

...was probably one /of the first. to come out with the notion 
that parents are teachers too, and that parents could do some 
things in their homes that were being done in the Headstart, 
daycare, and private nursery schools. The program was to P"'' 
together some positive interaction between parent and child. 
This' is not .to say that parents are pot positively interacting 
' with their children already, but that they could be trained 
to do some skill developme'nt and concept development, and in 

jout jthe child's educational ability and 



so doing, know more about 
the^way he learns and his whole mode of interacting. 
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As parents learn to be inore active participants In their children's 

learning, it is thought that they will assert themselves irore strongly 

in school decision-making. The developer points out: 

*'He hope thsit one of the functions that the training will serve 
will be to give the parents more control over the educational 
system— how to jnake it more responsive to their needs-.." 

The program is designed to fill a need for fasriljes above the 
income level required for Headstart programs but not affluent enough 
to afford nursery school tuition. The developers '^timated that three-^ 
fifths of the parents in the country fit t?is description. Because 
of the training component, however, one fan'^y will not be able to act 
alone to purchase "the product." Ho^ever^ any one of a variety of organiza- 
tions could set up a Toy Library. The Toy Library Program requires a 
course leader, a room for the weekly sessions, sane audio-visual equip- 
ment and storage space. Schools, daycare centers, public libraries, 
churches, industries and unions are among the target buyers. If no 
already-organized group in a community wants to take on the program, 
a group of par^ijts could incorporate as a nonprofit association "and begin 
to operate a Toy Library. 

The Toy Library program is meant to supplement the customary proce- 
dures for early childhood feducatibn. It established new roles for parents— 
the users, for organizations—the purchasers^ 4ind for the course leader— 
the facilitators. Furthermore, the developers contend that ^ physical 
products— the toys and other materials— are meaningless without the training 
program, vrfiich requires planning and money to set up. . 
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Smnaary 

The Toy-Lending Library Is a product «hich Is part oPthe Far Vest 
Laboratory's Responsive Prograa. The Toy-Lending Library, itself has 
two sub-products or components- These are (1) the toys themselves and (2) 
the parent training progrea. The relationship of these prograias and products 
EJay be seen as: 



FAR WEST LflBORATORT 
RESPOfiSIVE PROGRAM 



Other Prograass & Products 




^ Parent/Child Toy-Lending Library 
Parent Training Program -Toys 



You have now completed the introduction material for this siraulation- 
Turn to Task 1 , Preparing Guidelines for Developers of the Librarian's Manual 
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TASK 1; PttPARIMG fiUIOEHIIES FOR ttVElJOPEIS 
, OFTHE lIBRARIiM'SHHIIUAL 

Tine allowed (includina niading^ - 1 1/4 iMwrs 




« £ M 0 R AHD UH 

TO: Haabers of Teas Assigned to Prepare Guidelines for Developers 

of Librariajr*s ainuiil 

FROM: Doris Shin, Teas Leader 

SUBJECT: Preparation of Guidelines 

In the weeks ahead, our Parent Training ifatenals tea« is responsible far 
producing eight Resource Kits, one for each session of the Parent Training 
Prograa. Early in the Toy Lending Library project, a saaple Resource Kit 
was developed for the first session of the Parent Training Progra*. Although 
the results froa the testing of this first Resource Kit suggest a need for 
certain revisions, I think its foraat would be a good one fpr us to follow in 
developing the rest of the Kits. 

For the Resource Kit on tfie Feely Sag toy, we need to prepare the Libra- 
rian's Manual for the parent training session, a filastrip presenUtion, and 
instruction sheets for the parents in how to use the toy. The task of preparing 
the parent instruction sheets has been assigned to George Gregory, so the rest 

' of us need to work on the Libranan-s Hanual - 

I would like you to prepare a »eno to the team who will actually develop 
Section 2 of the Librarian's Manual. Section 1, which Is attached for you to 
read,.gives general background inforsiatlon. In contrast. Section 2, which you 
Should describe in your iww, will give step-by-step instructions that the 
Librarian can follow in teaching parents to use the toy. In order to develop 
this section, the team will need your to provide: 1) descrlptionstand. 
specifications the developer can follow regarding purpose, foraat, objectives, 
user (i.e.. Librarian) activities, constraints, and other relevant .Inforaati on; 
and 2) a brief outline of the main steps 1x) be followed by the Librarian during 
the Instructional session. 
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By raferring to the attached docuaents and those you have preyiously re- 
ceiyed> you should have all the infcniEticn you will ne«J to prepare this ancM- 



S£SX Zr FSSPAEISS CUIDELISZS FOR BEVSTOFEBS ~ 
CF SSB LIBSABIAS'S HASUAL 

This task calls jot the vrepcoKsHan of a memo vroindz^ng, giddelirjea for 
the develapevz of SecHcn 2 of the Libxadcn's Hcamdi for the parent trcdning 
session on the Feely Bag toy* Ion will have I 1/4 Jzours to coaptete it* It 
"should be no longer than the e<^t)aifnt of three (2) do^le^-spaced typed pages 
(300 words} ^ Ion ha^e heen provided with several doaumants (e.g.^ Feely Bag 
toy description and Section I of the Librarian's Hanmt—both attached to this^ 

a description of the Toy Lending Library^ a description of your role as 
a msber of the development team) whidi should provide you with the informa- 
tion you will need to accaiplish this task. Jou are free to re-read, refer to^ 
and use these docicrysnts th aiy way during this part and later parts of the 
siaulatipn. However, please do not refer to documents included in later tasksl 

E^najrber, you are not respcmsible for actually developing the Librarian's 
ManuaU Bather, you need to extract the relevant {reformation from the docu- 
ments provided and organize it in a rnem so i^voA the developaent tean will 
knou for whom lihe manual istoie written, in i^iat form it is to he written, 
what it ie sxx^yosed tc. acaoBiplish, etc. Jour moo should also outline Hhe ^ 
content of Section 2 of the manual^ i.e., -the steps the Librarian should follM^ 
iai the parent training session {e.g., introducing parents, leading rote-placing, 
eio.) 



flEWMDER. A description of the Feely Bag toy and the background infomatjon 
section (Section 1) of the LibraHan's Manual are attached. When you have 
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finished Task 1, please fill out a copy of the DDK Sinulation Task Evaluation 
Fora (goldenrod paper) before turning to Task 2. 
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FEELY SAG TOY 



EQUIPJ^ff: 

SmTI drawstring bag and two sets of cut-out shapes. Each set has 
tiiesc four shapes: a circle, a square, a triangle, and a rectangle. 

PURPOSE: 



to help the child recognize shapes by touch and by sight. 

RESPOWSIVT PRQSRAH PRINCIPLE TO 8E DISCUSSED IH COKJUNCTIOH WITH THIS TOY; 

Self-correction - The child can iaaediately find out the results of 
explorations, either froo the toy itself or fro« the parents. 

« 

6WE TO BE PUYED MlTH THIS TOY : 

One set of shapes are placed in the drawstring bag. Parents are instructed 
. V to shot* the child a circle, square, triangle or rectangle froa the second 
set of shapes, tell the child the naee of the shape, and ask the child 
to find a shape in the bag that is the saw as the shape the parent has. 



GENERAL IHSTRUCTIOHS FOR PARENTS: 

A. Ask your child only once each day if he (or she) wishes to play 
. ' the gaiae. 

B. The child nay change the rules^'of the^game at any time. . You must 
follc^ the child's rules if he/she changes them. 

C. You should stop the game when the child seeas to lose interest. 
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Section I 



HOH TO OPERATE A PARENT/CHILD TRAINING COURSE 



A great deal of planning has to be done i>efore a course for parents can 
be started. If all ti)ese plans are carefully followed, it will be possible 
to train many groups of parents to use the toys at hone witt their S-year-old 
or 4-year-old children* This part of the manual includes all the infonnatiOT 
that is needed before the. course begins. 

1* What Does the Course Try to Do? ' 

The Parent/Child Training Course teaches parents how to use toys and 
games that help preschool children learn some very important skills and 
ideas. In the course, the parents also learn soma basic things about 
.how children learn to think. 

Children roust develop the skills of seeing, hearSng, feeling, and so 
on—because these are the things that help them think clearly and well. 
They must develop their language skills because language is the basic 
tool for thinking. They must develop their ability to form "concepts'* 
(bigger ideas} because concepts help to organize thinking. They must 
develop problem-solving ability because problem-solving is the key 
purpose of thinking. 

m 

In the course, the parents also learn some ways of understanding 
how children feel about themselves > about their families,. and about the 
world. It is very iii^)ortant for children to: 

*^ feel good about thems.elves, their family, and their people; 

*^ believe that What they thinfeand say and do makes a difference; 

^ believe that they can be successful; and 

believe that they can solve problems. 

As the parents begin to play these games with the child, they will 
begin to take a more active part in ther child's education. At the 
same time, the parents will begin to make in^ortant decisions about 
what the child is going to learn and who is going to teach the child. 

2. Who Developed the Games— And Why? . 

For several years a grdup of men and women at the Laboratoiy for 
Educational Research and Development have been working on a program 
called "The Responsive Program." The Laboratory is a non-profit public 
institution that operates mainly with money from the United States 
Department of Health, Education and Welfare. But the money to work on 
this program came from the Carnegie Corporation. The people who 
developed these games have worked for many years with parents and child- 
ren and teachers. All of the toys used in the course have been tested 
and checked and tested again with many groups of parents and children 
in many different places. 
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The people who developed the games believe that: 

* children should not get any special rewards for learning some^ 
thing new— the only reward should i>s the pleasure the children 
get froQ the •act of learning; 

children should be free to "explore" what they ire going to 
learn; 

".children should learn^at their own speed; 

" children should learn right away what happens if they act in 
a certain way; and ' - j , • 

* things that children play with should lead them to discover 
other things in the world around thto- 

All the toys used in the Parent/Child Training Course were tested" 
very carefully to be sure tJiat they met these basic needs. 

3- What Materials are Heeded for the Course? 

There are eight Resource Kits available for the Parent Training 
course, one for each weekly meeting. The Resource Kits contain all 
materials (except audio-visual equijanent) which are needed to train 
parents in the use of one toy. Each kit contains: 

1. A Librarian's fenual for the course leader. 

2. A films trip presentation. 

3. A toy and set of instructions for each parent. 

When ordering the Resource Kits, be sure to specify the number of 
parents who will be in the course, so enough toys and instructions are 
Included in the Kit. 

Be sure to collect the toys and instructions from the previous week 
at each session so they can be returned to the Resource Kit and used 
again. There shou^ld be no cost to the parent if the toy Is lost or 
broken, but be sur& the parents understand that the toys should be re- 
turned so others can' use them. 

4. What Other Equipment Is Needed? 

The Parent/Child Course uses eight filrastrips (there is one for every 
toy) that show parents how to use each toy in several ways. With each,- 
, filmstrip a sound" tape Is used so that the filmstrips will seem to be 
''how- to-do-It" movies. 
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To show the filmstrips, the course leader needs: 

^ A films trip projector. 

One can be borrowed fro© a nearby lifaraiy or college or 
school, or church, or one can be rented fran a local dealer 
(See "Audio-Visual Equipment and Supplies" listing In the 
" yellow pages of tbfe local phone directojy). 

^ A cassette tape recorder^ 

One can be borrowed from a friend, or from a nearby 
library or college or school. ^ * 

The course leader laust learn how to operate these uachines before 
the first session of the course begins. The leader aust check electrical 
wall sockets. Alsp, the leader imist be sure to have a large piece of 
yfiiite paper taped on the wall or obtain a screen so the pictures will be 
easily visible. The filmstrip projector and the cassette player are run 
at the same time. One gives the pictures. The other gives the sounds. 

Where Can the Course 8e Located? 

The meetings of the Parent/Child Training Course should be held in 
the same place each week. A room may be tfound in any of these locations. 

School Building 

^ Church 

^ Library 

" Hospital 

" Vacant Store '- 
i ** Community Canter .f 
Town Hall 

It is very important that the location be within easy walking distance 
for the parents. A place near a local bus stop would be good, too. ' 

How is the Course Leader Chosen? 

The parents of. the coninunity will probably wmt to learn from someone- 
who lives in that same comnuhfly. ' They art not Hkely to be comfortable 
with an "outsider." The course leader shouTd be someone who is really a 
part of the community. The amount of money the course leader earns should 
not be very different from the amount earned by the pa'/«nts who take the 
course. But it jls not .necessary to be a mother or father to be a course 
leader^ A warm, friendly man or wonan— even one who Is not a parent— can 
probably do a good job. • ^ 
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7- How Are Parents Selected for "the Course?" 

If the course leader lives In the coraaunity. It won't be hard to 
find parents and ask them to cone to the weekly meetings. Haines of 
parente with very young children, can be collected by talking to the 
o^ice workers at the local eleaentary schools and high scliools. The 
Ic^al librarian ray have some names to suggest. If other coiraunity 
programs are already going on, the aides or parent coordinators may 
have good ideas. But, because schools already have to count ahead of 
time all the children who will be cooing into their kindergartens, the 
school office is the best place to get the biggest list of nanesof 
parents of preschool children. 

Parents cannot be signed up for the course by writing letter^ to 
them. Each parent nuist be told about the course personally. Getting 
a group of parents who are willing to take the course the first time 
Is the very hardest job the leader will have. For that reason the 
leader should run only one course at the beginning. Later on, with 
more experience, two or three courses could'probably be managed during 
the same period of time. 

Child-care during the course meetings will always be a-big problan. 
The child should not come to the meetings with the parent. Possibly a 
teen-ager can be given the job of taking care of some of the children 
so ttat the parents will have free time to learn. 

A local school district or cocrounity college may be willing to .give 
the parents adult-education credit for taking the course. The course 
leader should visit the office of the nearest adult-education program 
and show them this book. When the staff understands what the course is 
' trying to do for parents and chtildren, -it will be able to decide quickly 
if credits can be given. And the chance to earn adult-education credits 
should make more parents want to come to all_ the meetings once they begin. 

8. How Can Visitors be Handled? ' 

No visitors should be allowed before the third 6r fourth meeting. 
It takes time for -the parents to begin to feel comfortable with each 
other and with the course leader. But once the meetings begin to 
run smoothly, visitors who drop in should be made to feel welcome. 

If, by chance, a parent brings a child to one of the sessions, the 
course leader may be able to use that child as a "model" In the training. 

How do You Prepare for a Weekly Meeting ? 

f Here is a plan for the 8-week course. , . - ' 

During the week before the sesslorf, read the Librarian's Manual for 
that session.. The Librarian's Manual will explain what to do at each 
session and will offer suggestlpns-of what to say to the parents. 
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Before each session begins, be sure to set up and check the filmstrip 
projector and the cassette tape player. Be sure the tape is in the posi- 
tion where the sound track starts. Focus the filmstrip projector on a 
blank section of the wall or on a large piece of white paper taped or 
tacked on the wall. 

And, of course, be sure enough toys are ready so that each parent 
will be using the same toy at home during the following week. 

Finally, be sure there are enough printed instructions for each 
gaoe so that each parent can have everything needed to succeed. 

10. What Happens in a Weekly Meeting? ' ^ 

As the course leader you are responsible for setting up the activities 
of each session. The Librarian's Manual for the session should help you 
do thisJ Each week you will introduce a new toy and describe Its purpose<^- 
You can! then show a filmstrip presentation of a parent and child usirtg^-^ 
the toyl After the presentation, a kind of "acting" called "role^laying" 
is used: (a) to help each parent get a better idea of how each^ame works 
and (b) to help each parent begin to see what the child will learn from that 
special game. 

Role-playing means pretending to be someone else. It means getting 
the "feeling" of another person's part in the game. One person takes the 
other person's place and acts as he would probably act. Playing a^"role" 
lets the "actor" find out other people's ideas and feelings. (Children are 
"role-playing" when they dress up In adult clothes and pretend to be their 
parents.) 

In the course, the parent is really doing the same kind of thing— 
exceptjiow the^ adult is^T^ a child. In the^ourse, role- 

playing lets tjie parent imagine hew a child will act and feel in the game ^ 
' they will soon play together at hone. % 

As the course leader you will also need to answer questions which the 
parents maty have. The Librarian's Manual should help you fn answering these 
questions. At each session you may wish to schedule time where the parents 
' can discuss the toys and their experiences with them* 
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TASk 2: PJCPARINfi POR THE TRYOUT OF THE JCSOWCE Kit 

Please do not read this task until you have coiipletetf 
Task l-,«n^, an evaluation, fom for Task % 

TiJK allowed (Incl'jding. reading) - 1/2 hour. 
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«E«ORA,HDU« 

TO: Designers of Librarian's Hanual for Feely Bag Resource Kit 

FSOH: DoiHs Shin, Teas Leader 
SUBJECT: Tiyout of Feely Bag Resource Kit 

On the basis of the design for the Lifarsrian's Manual which I put together 
to incorporate the best features 6f all your designs, a protota^ of th^t manual 
has been prepared by other ffieisbers of sur^velbpoent teanji (I have attached 
the design and the prototype for you to lopk at.) Il is new tiiae foi- the Resource 
Kit on the Feely Bag toy (includfng the Librarian's Hanuil) ±o be tried out*y 
potential users. Our Research and Evaluation Team »nl1 conduct the tryoui. They 
Intend to tryout this Kit at at least tw different sites With a different 
Librarian For each site- They are going to have each Librai\ian attempt 16 recruit 
about six parents for the session. 

Even though the Research and Evaluation people are going to assume main 
responsibility for the tryout, we need to provide them With sonie information. 
First, given the small number of people with whom the matsrials will be tried 
out and our restricted travel fOnds, please describe briefly what you think the 
Research and Evaluation people should look for in establishing test sites, i.e., 

t 

what kind of sftes. Librarians and parents. A paragraph should be sufficient 
for this. 

Second, the Research and Evaluation team need to know what kind of questions 
we need .to have answered, in the' tryout. I- have been working on the questions for 
the filmstrip presentation and the session as a whole; I would like yoU to list 
the questions you think are important to have answered about tTie Librarian's. 
Hanual. List separately questions. that you would ask: (a) the Lib/arian, (b) 
the parents, and (c)'ari observer of the session* 

i 



TASK 2: TRSCVT OF UBSASUS^S KSM/AL 

In this task you are to: (I) describe vkat ike Eeaeareh and Evciuation 
Tecen should look for in selecting test siteS:, librariam^ and parents end 
(2) list the questions (sepa3pateU} for librarians^ parents^, and observers) that 
you vould like to have ansu^red in a tryoict of iSie Librarian's Manual that ve 
haoe dewloped^ 

The list of questions is the nore important of iSie ti:>o activities^ It 
should take no longer than the equivalent of tM (2) typewritten double^spaoed 
pages^ 

The description for the Research and Evc^ucMon Team skovHd he brief- 
one or ivo paragraphs* ' 

This task should take no more than SO minutes* 



REHIHDER. thfe design and prototype of the Librarian's Manual are attached. 
When you haTve finished Task 2 please fill out a copy of the DDI£ Slrmjlation 
Task Evaluation Form {goldenro'd paper) before tumifig to Task 3* 
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DESIGN OF LIBRARIAN'S HMUAL FOR FEELY 8A8 lOf SESSION 

The purpose of this sanual is to provide Instnictions to the librarian 
for leading the session of the Parent Training RrogruB vhich introduces the 
toy called FEELY BAG. This toy is designed to help the child recognize 
shapes by sight and touch. 

After reading the oanual, the librarian should be able to: introduce 
the toy FEELY SAG, run the filmstrip presentation, discuss the Responsive 
PrograiB's Principle Whidj is illustrated in the filastrip, lead the role- 
playing session, and answer questions the parents «ay have. 

The -Banual shojld be written at the reading level used in the first 
librarian's inanual. ITie format should be similar to the foncat of section 
two of the first nanual- The instructions should be double-spaced and ni^ 
bered. Any exanples which could be read aloud should appear beneath the in- 
structions, be triple-spaced and boxed. 

The librarian will need to refer to the manual during the session in 
order to follow the instructions and to run the filmstrip presentation- To 
help the librarian run the filmstrip presentation, a copy of the script in- 
dicating when frames are to be changed tiust be included in the manual . 

The instructions should provide information on:. 

1- Beginning the session by reintroducing the parents, 

' 2. Leading a <iiscussion about the parents exptr^tncxsr^yfiihthe toy 
from Session 1 (SOUND CANS). . 

3i Collecting toys and parent instrucjfcidhs from Session 1 (SOUND CANS). 

4. Introducing FEELY BAG and, its pose. . - 

5- Showing the filmstrip presentation. 

6. Discussing imporUnt points illustrated in the presentation; jMpha- 
sizing iha Responsive Program PrincipTe -Self Correction. (TJ« child 

^ .can immediately find out the result of explorationsji, either from 
the toy itself or from the parents.) ... : - . 



7. Distributing the toy FEELY BAG afld parent instructions. 

8- Leading-tSie role-playing session. 

9- Reviaring the purpose of the fEELY BAG toy. 

10. Ending the session with a question and answer period. 
The saanual should not be core than ten pages. It should be three--hole 
punched so that it can be placed in 2 ^all folder. 
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Section II 
SESSIOH II - FEELY BAG 

Instructions for the first training /session are listed belw. ^ 
Dcanples of what to say to the group of parents are provided in the boxes. 
You aay read these exsspleS aloud^ or use them to plan what you will say to 
the group* 

♦ 

1. Introduce yourself as^the course leader. 

For those of you who my hot already know me, mf name is 

, _. , I will be neetins with you. for 

the next several weeks to help you learn how tp use the toys we have 
here with your children;. 



2. Ask::the parents, to introduce theinsel^es. 

Since we'll be working together in learning about these tct^s ani 
and the ways children learn, it is lioportant that we get to knokr each 
other. Why don't we start by having each of you introduce yourselves 
to the, group? Let's begin (tq pry left, with the first ran, .etc.) 



3. Ten where the Toy Lending Library ortginited, and sihat it's puipose is. 



The Toy Lending Library «as developed at the Far West Laboratory for 
Educational Research by people yHio specialize in teaching young children. 
Hiese^peqple felt that children can and do learn in nany places other 
than school. They felt it is i^ortant for parents to take part in 
their child's education, and for children to have fun while learning 
things they will need to know. Each of the toys, in this program was 
designed to teach an important skill such as hearing, feeling, counting 
or pn*leffl solving. All of the toys have been tried out several tines 
with many groups of parents and children in raany different places. In 
the next several weeks you will be learning how to use these toys to help 
your children learn new skills and enjoy themselves while they are 
learning. 
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Describe >#hat iJie weekly aeetins Kill be 11k€ and what the parents will 
be expected to do. 

1 ~ ^~ 

At tte weekly meetings I will distribute a new toy to each of 
you. We ifin see a filinstrip showing a parent and a child using the 
toy, and then discuss how to use the tcy with your children. We will 
also have a chance to tryout the toys ourselves. After the jueeting 
you can take the toy home and let your chil<'^n play with it. The 
toy iDust be returned the following week so it can be used again with 
other parents. Hopefully we will have tiine to talk about the- experi- . 
ence you and your children had with the toy when you return it. After 
the eight iseetings you will be able to borrow other toys as you would * , . 
borrow a book from the library. Each of the toy%will have instructions 
with them so you will know how to play gaioes with your children when 
using the toy^ - - - 
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Introduce the tcry FEELY SAG and describe ite purpose. 

This toy is called FEELrBAG. It Is Bade lup of a s«all drawstring 
bag and two sets of cut-out shapes. Each set has these four shapes: 
a circle, a square, a triangle, and a rectangle- [SHOW THE PWIEHTS THE 
SHAPES AS YOU MENTIOH THEM] This toy was designed to .teach children to 
recognize shapes by touch and by sight. FEELY BAG should help your 
children learn to tell When shapes, are the saae and **en they are dif- 
ferent by looking at them or by feeling than. An Important aspect of this 
toy Is that It is self-corrective; the child pan f5nd out the results of 
his/her explorations Immediately either^ from th^ toy Itself or from the 
parents « 
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Show the filinstrip* 

Now I will start the f jlBstiip which shows how a parent uses F£ELY 
BAG to play gMes^with a child. £ 



Discuss iaportant points illustrated in the filostrip* (How to introduce 
a toy to a child^ what to do if a child makes up rules, v^at to do if a 
child picks out the wrong shape.) 

^ ^"^ ~^ ~ 

I think the filmstrip showed some important things to remember in 
platying games with children. If you recall » the parent did not begin 
by telling the child about the game they would play. Instead, she let 
the child play with the shapes by herself, fven after the parent tried 
to start the game she did not force the child to play. The little girl 
wanted to- play her own game jind the parent let her do juSt that. It is 
important for children to explore jwfhat they are going to learn and to 

enjoy learning. This is why the parent in the filmstrip allowed the 

* 

child to pT^ with the toys by herself. and did not play the game until 
the child wanted to. 
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Aiofter point the filnstrip illustrates is the use of specific 
' words when describing things to your child. When the girl in the 
picture chose the wrong shapes the parent said, "These two shapes are 
not the saro, try again" instead of saying, "That's the wrong shape." 

When the child did choose the right shape the par«nt said, "Yes, these 

\ 
/ 

two shapes are the sane; they are both circles." In both cases the parent 
told the child why the shape was right or wrong by using specific 
: ' words. 

Ycj will find some of these hints to use when playing with your 
child in the instructions which I will pass out with ths toys. 

NOTE: 

If the parents have not started asking questions or discussing the 
filmstrip, mention that they are welcoise to make conments at anytime by 
saying something like: . 

' It seems I've been doing most of the talking so far. I hope we can 
keep these meetings rather informal, so I want you to feel free to ask 
questions or make comments at^ anytime. Does anyone have a question or 
something they would like tb discuss with the group? 

* 
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8. If the parent do not have any questions, or if the group is. finished 

discussing tteir questions and concerns, give each parent a set of 

# 

instructions and a toy. 

9. Introduce the Role-Playing Session. 

Before youJplay the gaiae with your children, it would be good to 
practice with it. We'll divide the group into pairs and practice with 
each other. This practice Is called role-playing. Role-playing means 
that a person is acting the part of another person, the actor Is doing 
what another person would do in a real situation. Acting the part of 
another person helps you to understand how other people do things and 
what It feels like to he in "someone elses shoes." Each of us will get 
a chance to act or role-play the part of a chilid and the part of a 
parent. Acting the part of the parent shouldn't be very hard, but acting 
the part of a child may be hard at first. Try to pretend you are three 
years old arid your parent is showing you a new game. 

NOTE: , • ■ 

Before dividing the group into pairs, ask one jparent to role-play with you. 
Take turns being the child. After the others have seen what role-playing 
is like, ask if there are any questions, and then divide the group Into 
pairs. Remind them to let th6 child explore and to use specific words when 

talking to the child. 
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After each parent has had a chance to play the role of a child, bring 
them back together as a group, ask if they have any questions, and 
review the ganie FEElY BAG. ' 

O.K., How that we've all had a chance to feel what it is like 
being the child. It's get back together as a group. Did.you run 
across any problems trying to use specific words with the child, or 
letting the child explore the toy? - 

(If the parents feel like discussing their role-playing experier.ce, allow 
them time to do this. After they have finished, review the purpose of 
PEELY BAG.) , - 

I'd like to review FEELY BAG now, before we leave. It.'s a toy 
which is designed to teach children how to tell when shapes are the 
sane and when they ans different by sight and by touch. It should be 
fun for your children to play with the toy. AIIok theni to explore the 
toy, and to make-up rules and games. Remember to use specific words 
when your're talking to your children. If your child gets tired or, 
doesn't want to play anymore, then set the gaifie aside and ask the, child 



9 

to play another day. Don't force the child to play, because we want thesa 
to have fun when learning. 



11- End the session by asking if there are any questions. €heck tor see 
everyone has a toy and instructions • Remind the parents to read the 
instructions before playing the game, and to bring both the ins true-- 
tions and toy back next week. 



o 
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FSELX BAG SGilfT 



NAME 

XUHBER AUDIO 
1 This Is the first ga^e with the Feely Bag.* 



2 To play the gac:^, you'll need the Feely Bag and: two circles « 
squares, ti#o triangles, and two rectangles.* - 

-3 " The Feely Bag^ganes help children recognize shipf^ by both- touch 
* > and sight.* ^ : 

4 Begin by putting out the Feely Bag arid shapes.'* 

5 Let your child play withi:theB for a while. • ' 
(Child) "This a circle? I can't. Where is it?*'*- 

^ - 

6 To start the gone, put one of each shape In the b^.* 

7 " Put the other four shapes on a table or on the floor so your djild 

can s6e them.* ~ 

-8 Pick any.one of the shapes.* 

9 For example, yoa might pick up a circle and sty* ^ 

10 "Find a shape in the bag that is the same as this circle."* 

n . If she chooses * different shape, hold up your circle next to the 
one your child took out of the bag. 



12 Tell her, "These two shapes are ngt the same; try again." Aid 
put aside the shape- she took -out. 

13 (Child) "I try agairi-I'll find one yet— I'll findloinft yet- 
' find one.*"* , . 



14 -. (Child) "'ih-I found one already— "* - .. 

15 Since she did choose a circle, tell her. "Yes, these tw shapes 
are the sane; they are both circles."* 

16 (Silence)*v j' 

17 Sine? st:^ found the correct shape, put all four ishapes back into 
the Feely ^tg again. 

I 

18 Now pick up Another shape—for example. * triangle.* 



*Tliese mrks: shoM when the "tone" or "beep" is heard. Advance to the next 
frmi each" time at the tone signal . ■ 

' : ^ \ . 233 - 
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FRAME 

JIlfBER AlffllO 

19 Say to her, "Find the shape in the bag that is the sane as this 
traiangle.** 

20 If she peeks into the bag during the gaaae, tell her, *Wow find 
the shape inthout looking.* 

21 If she pulls out a triangle... (Child) "I found it." "These two 
shapes are the sane; they are both triangles."* 

22 Put all four shapes back into the bag..«then continue to play the 
ga»e.* * 

23 "This is a square. Find a shape that's in the bag that is the saae 
as this square."* ^ 

24. (Child) "I can't, 1 don't know where to find it." "These two shapes 
are not the «aiae;, tiy again-" (Child) "I'n trying to find it; here^ 
it is. I don't know — "* 

25 "These two shapes are the sanie; they are both squares." 

^ -> 

26 If your child t^iinks of new rules for tiiis gaffle^ let her play it' 
her way.* ^ ^ 

27 ' Here she's invented her own special way to use the Feely B|[g. 

28 Notice how the parent goes rijjht along witt- her, changing from one 
game to anbtter.* - " 

29 (Child) -"What's this thing doing in this for?"* 

30 Stop playing as soon as she seems to be bored or tired.* 



-END- 




TASK 3: RECCTtCMDIMS REYISIOife^ FOR THE UBPMm'S mm. 

Please do not read, this tiisjk iintfl you have completed ttsk 2 
^ -and an evaluation font for Task 2. . 

, lime allowed (incloding reading) r 1 hour 
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TO: Developers of the Teely Bag Librarian's Manual 

FflOM: Doris Shin, Team Leader 
SUWECT: Revising the Librarian's Ifenual 

I have just, received the report on the tryout,of the Fsely Bag Resource 
Kit^ You will find a copy of the report attached to t5iis wmo. 

Thers are tj*o things I would like you to do with the infomation in 
this report. Hie f^fst and aost iaportant is to list your suggestions for 
revising the Lii>rariin'3 tenual based on the results of this tiyout. 

The second is- to entique the tryout by aniswering the following questions 

1. What are the najor strengths of the design and iapleaentatipn 
of tije tryout? 

2. What are its' BJajor weaknesses? . 

3. If the' Research and Evaluation Teaa were to undertake a second 
tryout of this resource kit, what additional instructions would 
you given theo regarding: 

"selection of test site(s), librarian(s5, parents? 
"fijaking arrangements for: the tryout? j 

*=directions which should be given to the librarian(5) before 
the session? . - 

"specific directions and questions for the evaluator observing 
the session? : 
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TASK 3: ESCCIKSEDISG SSVISICSS FDS 22? USSASUiS'S HASVAL 

i ^ 

^jare are too cciivities intiptved in iSiie task. Listed in order of 
importance tkey are: ' ' . 

\ i 

Z. EecoBtsending revisions for iSie Librarian's Manual. (This product 
shoitld he no tonger zhan the eqptU>almt of iSiree (3) dqUbte^^aoed 
tzfped pages») i 

2- Critiquing the tryoiit. (5!!kis product shoiUd he no longer fhan i32e 
equivcilent of izDo (2) doitbte-spaced ti^ed pages*) 

You ha^ 1 iiaur to corrplete this task. 



REMINDER- The Swamary Report of tl>e tryout iAich you will need io accosplish 
Task 3 is attached- When you have finished Task 3 please fill out a copy of 
the «)*E Simulation Evaluation Forra before turning to Task 4- 
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smm OF THE mm of the feely bag tssmcE kit 

This unit k3s pilot tested at the saae ti^o sites used to test the prototype 
lesson: the fieighbortiood Care Center in the San iFrancisco Mission District and 
at the West Oakland Hursery Sdsool. A msber of the staff was recruited at leach 
organization to serve as tte. toy librarian. This person agreed to recruit six 
parents to atterid a toy training session. 

Heighborhood Carg Center*. This Center is located in the Missian District, 
an inner-city, oulti -ethnic neighborhc^od. The Neighborhood Care Center has 
j)een in operation for three years, triih funos supplied by lirAtti Crusade, federal, 
and ciiy sources. 

Mrs. Lopez, who is one of three part-tinse paid staff saeobers (paraprofessicnals) 
and has been with the Center for two years, was recruited to serve as the librarian. 
She was faniliar with the project since she had served as a "parent" for the 
prototype test. The materials (librarian's guide, AV equipacnt, fil«strip, tapes, 
toys, etc.) were delivered the day before the scheduled session. We helped 
I4rs. Lopez check the material and determined ttat she Jcnew how to operate the AV 
equipment. 

Mrs. Lopez reports that she spent two hours reading arid preparing for the 
training session, which was scheduled for the following evening at 7:00i).nj. 
I arrived at 6:30. Mrs. Lopez had already set up the equipisent in one of tiie 
jfplayrooms. By 7:15 five parents arrived. ^{The sixth, had called to say she 
couliln't come). Two were Latino, two were Black and one was Anglo. All had 



the Cfensus Bureau statistics show the 1970 population of the area as 3235 Anglo, 
48X Spanish sumane, 5% Black, 2< Indian, 10% Other Specified (generally, ^ 
persons from the Pacific incljiding Samoa, Ouam and the Philippines), and 
.31 Other. 
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children enrolled in the school. They obyiaisly knen Hrs. Lopez and each other. 
The session lasted from 7:20 to 8:30. By arrangenent, Mrs. Lopez seryed 
refreshiDents while I held a brief interview with each parent individually. 
After the parents left, I interviewed Krs- Lopez. 

Results. Krs. Lopez followed the reccaraended sequence to the last detail, 
except that she had to explain what the Tcy Libraiy was, ttat this was a try- 
out and why I was there. This would nonually haye been handled in the firet 
session. We should have anticipated this probleo. Suggest we produce a very 
short orientation for tiie librarian to use whenever any of theisanu^s 
are tried out with a new group of parents. This wouldn't be encountered in a 
regular series of sessions but was a problen] at this site. Fortunately, Mrs. 
Lopez handled the situation adequately. , ^ 

The filmstrip presentation got good attention and seeined to serve as a use- 
ful orientation. Getting parents to handle toys theznselves worked out well. 
But the role-playing was a disaster on the first two trials. Neither of the 
parents could get the idea of playing a child's role. Finally, it worked ^hen 
Mrs. Lopez played the child's role with one of the participants playing the 
parent. I 'in not sure whether there is a language problem, a cultural problem, or 
whether they needed n better model. (Say, a filmstrip depicting role- playing.) 
Best guess is il^s language and jsodel. Both of the parents who played the child's 
role said they didn't know vAat was expected of them. 

The interview results indicate that all five parents were favorable to some 
degree. All of them could explain the purpose of the game and each wanted to 
try it with their children. Howevelr,.. only two of the five parents could describe 
the Responsive Program Principle cdvered in the lesson in their own. words, 
even after 1 gave one example. * 

» e 

2a0 , ^ : 
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Mrs. Lc>pez said she thought the Instructions were easy to read and easy to 
understand. She thought the filnstrip could have lasted longer. She said tl?at 
she had never heard of role-playing and that the ttescription in the sanual Has 
too short to give iier a very good idea of what to do. She would also like to 
see a "question and answer" section so she would have better answers for sone 
of the parents questions- 

TTiis seeias to be a good idea. Questions asked at this session: 

° Are there other gaiues to play with the toy? 
iftat do I do if a piece is lost or broken? 

" How long should I play the garae? 

Vtest Oakland Nursery School. The school isa cooperative located in a low- 
incoBie area in Oakland * which has operated for five years. «rs. AJler, who has 
two children at the center, was recruited. She was interested in the idea but 
unsure of her ability to act as the librarian. The advance preparation was simi- 
lai: to that reported above for the Neighborhood. Center. Only four parents appear 
ed for the 10:00 a.in. session. Twp white and two Black. 'Because of our experi- 
ence at the Neighborhood Center and at Mrs. Adler's request, I spsht about 10 
minutes explaining the toy library project and the .purpose of the tryout. 
(Recognize this was not what was planned, but it seemed necessary to set the 
stage and put Mrs. ;Adler at ease.) 

Mrs. Adler started by giving each parent a set of tJie toys and then she 
talked about them for nearly fifteen minutes. The parents got restless. Finally, 
when the filmstrip was shown, interest picked up. Had Mrs. Adler followed the 

*0akland Public Schools information indicates that this neighborhood is predomi- 
nantly Black (approximately 65%) with some white (approximately 25X) and the 
remainder Spanish Surname and Asian.* All schools in the area qualify for 
Title I ESEAjnonies. 
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suggested schedule^ i think this could have been avoided* Ttole-playing session 

again got off to a slow start. D)uldn't get first two parente to play either 

role. Again, as in San Francisco, it wor^^ only when Mrs. Ad]er played the 

t 

child's part. 

i. 

Hrs. Adler skipped coverage of the Responsive Program Principle, so I was 
unable to detennine if liiis part of the session worked. All four parents 
were interested in the toys, knew how to play the gare, and said they Intended' 
to use theni with tiieir child. 

Note: I have called all nine parents lAo attended at the two sites one 
week later. Seven of the nine had used the toys with their children* 

2~Haven*t had tline yet. 

3~Played once. 

, 2~Played twice. . . ^ 

'1 —Played four times. 
1— Played six tiines. 

Average tiine played— "about ten nrinutes." 

The following table sumnarizes responses to key questions in the follow-up 
interview with the nine parents. (Please refer to Table- 1, page 7.) 
Conclusions 

: 1. Test plan failed to anticipate tfef although this was a test of a sub- 
sequent lesson, it would be a new experience for the test subjects. 

2. Jhere were major differences in the teaching skills of the librarians 
recruited at the two sites'. " t 

3. Both librarians reacted positively to materials. Both liked the film- 
strip, but both thought it could be longer. ^ 
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Nunber of Parents 



TABLE 1 

SESPOKSES TO KE¥ QUESHOKS 
(Fo11oH-ll^ Interview) 



Saru Francisco 



Oakland 



Value of Session. - 

(a) Very useful —quite worth attending. 2 

(b) Useful. 3 

(c) Not so useful— Could have done . 0 
something else with njy time. '* 



0 
2 
2 



Did librarian help to learn how to use the 
toy? 

(a) yerx helpful. 3 

(b) Helpful. 2 
' (c) Not viery helpful. 0 



0 
3 
1 



Did the fllmstrip help to learn how to use 
the toy? V 

(a) Very helpful. 3 

;(b) Helpful! - . 1 

(c) Not very helpful. 1 



3 
0 
1 



Would you) attend another session to learn 
. how to use a new toy? 

(aXYes' ■.' - ' ^-^ 

(b) Ho . ' * 



3 
1 
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4. Role-playing did not work well on either site. 

5. The Responsive Prograa Principle was presented at only one site, and 
there the results were disappointing. 

6. All the parents learned how to use the toy and how to play the gasie. 

7. The majority of the parents report that they have since played the 
game with their children. Seven of the nine parents said they would 
attend a second session. 

8. Although it is hard to support on the basis of only three tests (these 
two and the prototype. test), it appears ttat the present model of 

the lesson is adequate only in the hands of a relatively experienced 
.teacher (e.g., Mrs. Lopez). The present form of the lesson was not 
adequate for Mrs. Adler, *rfiich suggests one of four things: 

(a) The problem is pecujiar to Mrs. Adler and the Oakland test - 
site. (Hot a valid problem.) 

(b) "Librarians" should be seljy:tM the Jjasl^^ — ^ 
teaching experience. (Selection problem.) 

(c) iTl^experlenced librarians must be used, they should be trained. 

{Trairtingi3roblem^iand"we^ren^1r-funded to- develop^raining — 

except for a self-Instructional package.) 

(d) If inexperienced librarians must be used and -we can't provide 
traininj, other than by a self-instructionST package, we need 

to strengthen the model— e.g.,, more examples, longer description 
of principles, more concrete suggestions for activities, question 
and-answer sections, etc. 

9. This lesson and the protoiype seem to be moving in the right direction, 
but we haven't developed a model (yet) thit fully meets our design 
requirements. , 
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TASK 4: 0UTLiKIH6 A SOtlPT - 



Please do not read this task until you have 
completed Task 3 and an evaluat'bh fqm for 
task 3. 



TiBe^ aUowd (incliKlijig reading) - 1 hour 
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MEMORANDUM 

TO: Developers of the Librarian's Manual on Feely Bag 

FROMi Doris Shin, Team Leader 

'SUBJECT: Producing a Sound Filmstrip to Model Role Playing. 

After reviewing the Summary of the Tryout of the Feely Bag Resource Kit, 
which you reviewed in suggesting revisions for the Manual, it is apparent -that 
the role-playing did not work welj at either site. I think we need to produce 
an audio-visual aid depicting role-playing, vfith parents playing both the 
parent's and the child's roles. Undoubtedly a color movie would be effective, 
but it's beyond our development budget and would also make the price of the 
Toy Librarian's materials too high. I think we could afford a sound filmstrip 
if it's kept "quite short. The challenge will be to produce something that 
carries the necessary action in both the audio and the visual components. Per- 
haps if we had a rough script it would give our staff something tangible to 
consider. 

As you know, Tve been working on the other AV production, but I don't " 

„ _ ^ • ^ 

have time to work on this. Please" see what you can do about drafting an 
outline of a script. Use the Feely Bag as the toy. You'll probably want to 
start with a parent training session in which two parents are assigned roles 
of "parent'^ and "child" with the "parent" ^Aeaching the "child" the game, and 
then have the parents reverse roles. You might want to use th^ Feely Bag Script 
attached to the end of the prototype Librarian's Manual, which I gave you 
before we were preparing for the trydut, as a source of ideas. (See task 2 
attachment, last 2 pp.) Remember the purpose^of this filmstrip is to help 
parents in role-playing so that they and their children can get rtore out of 
the Lending Librai7 toys. ^ 

' . ' ' - 245 
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I don't need a complete shooting script, but I do need stnae concrete 
ideas frotn you about what the filmstrip might look like* Please be sure 
to include your ideas about the visual content that should accompany the 
audio. The script you're outlining should be under five minutes in duration 
(and probably not more than 50 film frames). I know this is a tough assign- 
Bient and a very short notice. But see what you can do- 



: TASK 4: 0UTI37iISG A SCRIPT^ 

This is a difficult task sinae you will he given only t hour to work on 
it. Your challenge is to use the time as productively as you cim. A oorrplete 



shooting script is not required. Do try to outline tTie entire content of the 
sound filmstrip and to produce some detail (both audio aod visual) for 
first severed frames. - ^ 



AAA 



REMINDER. When you have finished Task 4 please fill out, a copy of the DD&E 
Simulation Task Evaluation Form (goldenrod), then complete the General Evalua- 
tion Fora (blue). When you have done this, please check to be sur6 you haVe 
completed the entire Battery (Biographic Form, Self-Ratings of Performance, 
Self-Ratings of Knowledge, Knowledge Test, 4 Simulation tasks and Simulation 
Evaluation Forms), and then send the completed Battery with your signed 
contract in the return envelope provided. 
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DDS£ SIKUmnOS TiiSK EVALUATION FORM 
Task fio. ♦ • 10 ilo._ 



s 



1. How long did you take to c(»?)lete this *task?_ 



2> How satisfied were you with the dpcumentCs) you produced? (Check one.) 
fj Highly Satisfied £7 Satisfied £7 Undecided 

£7 Dissatisfied /T Highly Dissatisfied 

3- Have yon ever d-^ne a task like this before? fJ Ho fJ Yes 

If yes, how mr tiiaes ? 

Hhen was the nost recent time (date or approximately how long ago)? 



1 4. Have you ever received any training (courses or op-the-job) in how to 
perform a task such as this? 



U Ho LL Yes 

# 

If yes, please brieflv describe when^nd where, 



5. Hhat modifications would you reconmehd to improve ttis simulation task? 



FRir 



DDiE/CAB/4-75 . 

1975 Far West Laboratory for Educational Research and Development 
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Scoring llespbsises cai the DOGB Xoowle^e ^^\est 
Loizd Bladcvell 
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SCORING i£SPOXS£S QH THE i»&£ KKOtfLEDfiE TEST 
- • 

Although it oay be difficult for experts to agree on the kooirledoe or 
cocpetencies that Sire involved in educational IX)I£, i#e have tried to design 
questions «iat can be used to assess knoMledge and (especially} ability to 
apply knowledge which He feel would be important for inost DDI£ practitioners 
regardless of their specific content areas. 

Scoring these questions irill involve sooe Judgaents by the scorer re- 
garding criteria and their attainaent; in the following pages we have sug- 
gested criteria and saethods of scoring that cay help these judgeents and 
Bay facilitate agreeiaent anong scorers- However, ypu sight want to adopt 
your own criteria and scoring systeo to fit the particular needs or aiiphases 
of your program or situation. 

For each of the ten questiorts on the knowledge test, we have suggested: 
1) a checklist for responses or categories of responses, 2) criteria and 
isethods for scoring the adequacy of responses, and 3) exanples of responses 
that demonstrate application of a broad scope of relevant knowjedge to the 
questions as a guide to determining a ''scope" score. 

^ The "response checklist" for eadj question lists those cpnponents of a 
response that we feel are iinportant enough to he counted toiftrd a rating 

of adequate. The '^suggested criteria and aethods for scoring" indicates 
-'4 - - 

Hl|ich and how many of the components listed in the checklist we feel are re- 
quired for a subject's response to be judged "adequate.** That section also, 
^presents a method for deriving a "fraction score" for degree of adequacy. Al- 
, though you may want to rely solely on a rating of "adequate" or "inadequate" 
for each response, we feel that a "fraction score" indicating bow nearly a re- 
sponse approaches adequacy would also provide useful feedback to subjects. Sifg- 
gestions for deriving a third score— the scope (of relevant knowledge applied to 
a question)-- -arc also provided in this sect^'on* Whereas the other two score in- 
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dicate hon a response coc^ares to nininx^ requiraoents for adequapy» the "scope 
score* is independent of these requireswits and reflects the "breadtt" or di- 
versity of relevant knowledge applied to the question. For each question » scnae 
responses thit laijht be Judged to demonstrate the application of a broad range 
of relevant knowledge are suggested. A scoring scale is presented under the "scope 
score" section for question #1 as a guide for assigning the "scope score." At 
this point, we don't feel that standards or requirefients for €he score can be pre- 
detenained so we only present suggested guidelines forrwhat will be a veiy sub- 
jective judgaent on your part. Despite this Subjectivi^, we feel that the scope 
of relevant knowledge applied is an i^wrtant aspect of coapeterice and should be 
included in assessisent. 

Although you Eight want to develop your own criteria and/or scoring systea^ 
preliminary field testing' indicates that our suggested scoring methods have some 
validity: knowledge test total scores for adeouacy of response and scope 
of response were only laoderately related to each other, but they each discrimi- 
nated between students and professionals and were related 'to total scores for 
adequacy and scope on the simulation exercises. 

Although you would probably «nt to refer to the scoring manual while 
scoring a subject's response to reaind you of criteria and siethods, it would be 
much more efficient and. econojjical to use a separate sunmary sheet (like that on 
the next page) to record the scores, since your evaluation of many responses 
can then be recorded on one page. It is also suggested that all subjects' 
responses on one question are scored before proceeding to another question. 
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QUESnON 1 ; PRQgJCT OESGtiniCK 

If you yttre given the task of developing Uib product describe] be1ow» what 
Icpor^t infonDation would you need that has not been included? 

The "yictorian &igland" a>urse will serve as introduction to art, 
literature, and social life in Sigland durinj the Victorian period. 
The course will include selected readings fron texts on Victorian 
social life and the arts,, but will eaphasize the visual arts and 
literature of the penod as instructiopL in those areas as well as 
- sources of infoniati on about social ofstcms. "Victorian Digland,*" 
when produced and packaged, will include the following: 1} ^ book 
of readings, for each student, including works^by and about resi- 
dents of Victorian England, and 2) a paperback book for each stu-- 
dent, of black-and-white photographs of Victorian painting^ sculpture, 
and architecture. *\ * 

Users will be requimJ to read the required text, but user activities 
will focus on analyzing and discussing reproductions of Vicioriah art . 
dnd several of the best-known works of Victorian authors. 

A. Response diecklist ^ 
objectives 

user group 

B. " Suggested Criteria and Methods for Scoring 

I. Rating of ''adequate" or /^inadequate." 

In order to receive an "adequate" rating, the subject lausl^^receive checks 
for' "objectives" and "user group." -' 

II . Degree of adequacy score— sa)re one point for each of the responses on 
the checklist above Included In the subject's response, (In order to 
obtain equivalent scores for each question on the knowledge test, you ' . 
will probably want to convert the point scores into scores on a standard 
scale, e.g., one^jf two possible poin|s on question 1 would be the equiva- 
lent of 5 of 10 possible points and so' would receive a score of 5 on a 
0-10 scale, as would two points of a possible four on question 3.) 

III. Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (very lisiited to 
very broad). This score is independent of the adequacy score—a 
subject's response could be judged brosd in scope even though it 
did not Include all the required aspects. You alight want to' use 
the -following guidelines to help assign this scope score. 
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0 - included nothing other tJian responses required 

1 - included one or tm responses which i#ere not * 

required and Which you judged to be acceptable ; 
but not especially isiaginative or indicative of 
a broad scope of relevant knowledge 

2 - included a great* ni!ri>er of responses which were- 

not required and lAidi you judged to be acceptable 
but not e^ci ally imaginative or indicative of 
a broad scope of relevant knowledge 

3 - included one or two responses which were not re- 

quired and asay not even have been on the checklist 
)riiich you Juc^ed to be acceptable and very inagina- 
tive or indicative of a broad scope of relevant 
knowledge . * 

4 - included sany responses rfiich were not require and 

nay not even have been on^the checklist which you 
jud^^to be acceptable andl very inagi native or 
indicative of a broad scope of relevant knowledge 

b) exanples of possible imaginative responses indicating application 
of a- broad range of relevant knowledge 

Evaluation plan, teacher training necessary, entry level require- 
inents, costs and other constraints, nxxJe of instruction^ inter- 
farcing with existing prograias. 

Saraple Responses and Suggested Scoring 
i- Adequate sample response 



V 



formal set of behavioral objectives 
cost of naterials 

age level ^of subjects intended to use" materials 



Since this response includes both criterial Items on the checklist, it 
would receive the nsaxinum possible, two points ("lO" on the 0-10 scale) 
and a rating of "adequate," but would only receive a "1" on the 0-4 
scape scale because one acceptable but not required con^wneht ("cost 
of materials") which was j'udged to be not especially Imaginative or 
indicative of a broad scope of relevant knowledge was included. 
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Inadequate^ but broad scope saiiple response 



target audience 
mode of Instruction 
duration of course 
estiisated price of package 
s3ethod of dissemination 

Since "objectives" (crucial information for a developer) was not included, 
this response .received only one point (a score of "5" on the 0-10 scale) 
and was Judged Inadequate. However, it received a scope score of ''S" for 
the scope of knowledge indicated by some of its other coK?>onents (especially 
*Vnode of instruction"). 

/Uiequatf and broad scope sample response 



learner characteristics, i.e., age, ability, SES, year in school 
placement in curriculum, i.e., preceded by and followed by which courses 
preferred delivery system 
anticipated outconies/objectives 
availability of different media 
constraints of time and money 

Adequate; (score of "10" on 0-10 scale)i "4" on 0-4 scope scale. 

^ * 



1 
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QIIESTION 2: OBJECTIVES 



^ jQn the basis of tte following description^ specif appropriate teniinal ob- 
jectives for rthls product- ^ 

The "Plant Life of Hew Zealand" filia MilL incl4ide-stiM^photos^^ -- 
as well as siovie sequences of New Zealand's Most conaon plants both > 
wild and domestic. The filin will be divided into several segwnts, 
eat4) covering-a different family of plants. Each segM»t will be 
followed by a sjiort quiz -oxi plant Identification and associated know- 
ledge. The fflinis intended to faniliahize the high school biology or 
ecology student with New Zealand's coanon plants and their fanilies and 
teach the student what plants are edible and what plans are poisonous. 

A. Response Ch&cklist 

_ l^'-^^^tom ' 

behavioral (i-e., specify what the student should be able to dto) 

II. Specifications 

conditions under which the behavior will be accomplished (e.g., 

given photographs of ) . . 

criteria for scoring student performance (e.g., 8GX of' plants 



correctly identified) 
III- Content 

identify plant families or meobers of families 

/ 

' distinguish between or identify poisonous and e44ble plants 

B. Suggested Criteria and Methods of Scoring 

I. Rating of "adequate" or "inadequate." 

In order to receive an "adequate" rating, the subject must receive one 
check for each of the three categories (I.e., "form,** "specification," 
and "content")^ 

II. Degree of adequacy score—score one point for each check with: a maximum 
ofM)ne counted for each of "specifications" and "content." (Convert 
point score into equivalent on 0-10 scale.} 

Ili. Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (from very limited 
to very broad). You might want to use the guidelines syggested 
under question #1. 
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b) exan5)les of possible imaginative responses indicating application 
of a broad range of relevant knowledge 

Give characteristics -rfhich distinguish plant families or poisonous 
and edible plants, uses of plants, repeat quiz until achieve lOOS 
ident+ficatlon level. 



Sample Res'ponses and Suggested Scoring 
I. Inadequate sanple responsB 



students will be able to identify and distinguish edible and poiso- 
nous plants conmon to Hew Zealand 

students will be able to classify plants common to New Zealand 
according to family 



Since there are no conditions (given pictures of plants, real plants, 
. one plant at a time, names of plants) and no criteria for 4eternnn- 
ing attainment of objectives, this response received 2 of 3 re- 
quired foints^Ca score of "7" on Ihe 0-10 scale) and was judged in- 
adequate. It received a *'0" for scope. 

II. Adequate and broad scope response 



students will be able^to: 

recognize and identify by name and family the 'conmon New Zealand 
plants from their pictures 

suiiinarlze the major characteristics of the common New Zealand 
plants given their names 'v^. 

identity the common New Zealand plants given their major charac- 
teristics , . 

describe the environment in which various types of common New 
Zealand plants grow 

distinguish between edible a. -i poisonous common New Zealand 
plants upon encountering them ' ^ 

attaliwent of the above objectives are subject 'to a prerselected 
criteria of satisfactory performance | 



Adequate; ("10" on 0-10 scale) since has form, specifications, and content; 
"4" on scope for objectives a6out environment, characteristics, and identi- 
fication given characteristics. 
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QUESTION 3: jgPECTS OF TRYOUT 

An instructional filsa on the" behavior and attitudes a woman might adopt 
during pregnancy to help her prepare for chj Id-birth has been developed 
by a ccomunity health service to be used in""conjunction with reading, 
discussion, and practice in a "store-front" weeWy course for pregnant 
wonen. In order to try out this filn on as many ^oplfe with as few- 
actaini strati ve problems as possible, .the de^^yelopers plan to show it to 
several hygiene classes in a nearty high sdiool and then test for know- 
ledge as well, as interview for attitudes toward the film- 
On the basis of this description, discuss* briefly at least four aspects 
of this proposed try-out that could affect the relevance and iuportance 
of the Information it would yield. 

A- Response dieckl-ist 
I- User -differences 

nsotivation (e.g., pregnatnt/non-pregncnt) 

education 

attitudes 

age/sex/race/SES 

other 

II- Setting differences 

storefront vs- cTassrown 



III- Resource differences 

. , e quipnient/comfort 

IV- Use differences 

_not as part of on-going weekly course 

not in coniunction i^ith reading, discussion, practice 

V. Method of assessment differi»nces 

quiz and interview rather than observation of behavior 

B* Suggested Cr*iteria and Methods for Scoring 

- I. Rating of "adequate" or "inadequate." 

In order to receive an "adequate" rating, the subject must receive 
at least '4 checks which must include one check for "user differences. 
A maximim of two checks can be counted for any one category. 
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II. Degree of adequacy score—score one point for each of the checks counted 
toward the total of four required for an "adequate" rating. (Convert 
"to score on 0-10 scale.) 

III. Scope (of applied relevant knowledge) score 

„ a) rate the subject's response as to the scope of relevant knowledge 

brought to bear on the question on a scale of 0-4 (from very limited 
to very broad). You might want to use the guidelines suggested 
under questioirf #1 . 

b) ^examples of possible imaginative responses indicating application 
of a broad ?ange of relevant knowledge 

Captive vs. volunteer audience, resenfanent of parents of high 
school students, test-taking "set" of students, sex of teacher, 
authority-based relationship of teacher and student vs. service- 
based relationship of health personnel and "clients," 

Sanjpla Responses and^Suggested Scoring 

I. Inadequate sample response 



discrepancies between the intelligence, age, knowledge leveT'of the 
high school sample and the storefront intended users 

most high school istudents will not be pregnant 

the class views the film but has no reading, discussion, or practice 



Since this response indicates fpur potentially important user differences 
and one "use" difference, it was judged too narrow to be adequate (only 
two points maximum can be counted for any one category) and received 3 
points of 4 possible ("8" on 0-10 scale) for degree of adequacy. It re- 
ceived a "2" on the 0-4 scope scale for Its two additional user differences- 

II. Adequate sample response . . 



a) most hygiene class students will not be pregnant and so will have 
a different attitude toward the film than would pregnant women 

b) hygiene classes are mandatory wTiereas pregnant women are in store- 
front course-from choice 

c) storefront courses include reading, discussion, and practice which 
hygiene class does not 

d) try-rOut is not really testing for attitudes or behaviors 
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This response includes a user difference (a), a setting difference (b)^ 
a use difference (c), and an assessment difference (d) and so was judged 
adequate ("10" on 0-10 scale). It received a scope score of "3" primarily 
for (b) and (d). 



QUESTIOH 4: JUDGING REVISIONS 

List three of the basic criteria for judging the adequacy of revisions which 
have been tnade in a product. 

A. Response Checklist 

effectiveness (e.g., problem solved, greater user satisftction) 

cofflpatibility (i.e., degree of fit vrith other parts of product) 

impact (i.e., da other parts of the product have to be revised 

becaus?of the revision) 

^ " feasibility 

are there better alternative revisions 



based on research or evaluation data 



B. Suggested Criteria and Methods of Scoring 

I. Rating of "adequate" or "inadequate." 

In order to receive an "adequate" rating, the subject must receive 
three checks total whick must inclijde one, and may include two, for 
"effectiveness." Two checks may be counted from any category. 

II. Degree, of adequacy score—score one point for each of the- checks 

counted toward the total of three required for an "adequate" response. 
(Convert to score on 0-10 scale.) 

III. Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (from very limited 
to very broad). You might want, to use the guidelines suggested under 
question #1. 

b) examples of possible imaginative responses indicating application of 
a broad range of relevant knowledge 

Is there evidence that improvements are really due to the revisions, 
are revisions based on the field test results? 

" C. Sample Responses and Suggested Scoring 
I. Inadequate sample response 



(do people use the product better 
do they like it better 
' do they meet the performance objectives -better 
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All the cofliponents of this response involve "^effectiveness" criteria, 
so it was too narrow to be adequate and received two of the three re- 
quired points ("7" on 0-10 scale)* It was scored "0" on scope. 

II. ^Adequate and broad scope sample response 



a) Do revisions solve profajcm which initially njbtivated faaking 
revisions? . • 

b) What is the evidence for detenninirig the answers to the first 
question? 

c) Is 'tfie overall product consistent internally (relation and 
balance between parts)? 

d) Is the overall product consistent externally (i.e., does It 
^ conform to the objectives originally set for it)? 

e) Are revisions identified for easy reference* so they can be ^ 
compared to the original versions? 



This response has co.nponents involving effectiveness (a) and conipati- 
bility (c and d) and so was judged adequate ("10" on 0-10 scale). It 
also received a score of "3" for scope/ 
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- QUESTION 5: nUHSTRIPS VS, HQVIES 

Briefly cocipare sound flwvles to sound filsnstrips as instructlcmal nedia 
from the viewpoint of the developer s the distributor^ and the user; list 
sone advantages and^ disadvantages of each. 

A. Response Checklist 

I. Feasibility 

expense (i.e., nwvies niore expensive) 

^ cost and availability of equipment needed to produce 

(i.e., covies require more) 

availability of equipment needed to use (i.e., fil»strip 

equipeient is less available) 

II. Iinpact 

movies core effective if dynamic action require* 

• iTiovies laore effective if facial expressions important 

filiastrips usually as effective as iMvies except when 

dynamic action or facial expressions important 

filmstrips are more easily edited 

filmstrips can be more easily stopped and adjusted to pace 

of students, they provide more opportunity for teacher input 



III. Inclusiveness 

Advantages and disadvantages of movies and filmstrips listed from 
viewpoint of: 

'User 



devel(V,3r 



distributor 



B. Suggested Criteria and Methods for Scoring 

' I. Rating of •'adequate" or "ihadequate." 

Irr order to receive an "adequate" rating, the Subject must receive 
at least one check for each of "feasibility? and "impact*' and all 
three lines checked for "inclusiveness." 

II. Degree of adeguacy score— score one point for each of the checks 
counted toward the total of five required for an "adequate" rating. 
(Convert to score on 0-10 scale.J^^ 
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III. Scops (of applied relevant knowledoe) score 

a) rate the subject's response as th the scope of islevant tawledge 
brought to bear on the question on a scale of 0-'4 {froa very 
limited to very bread)* You jsi^t want to use the guidelines 
suggested under question fl . 

b) exanples of possible isaginative responses indicating application 
••^ of a broad range of relevant knowledge 

Hovies listed in ibre catalogues, filrastrip package nay become 
separated, filnjstrips njore individually tailored/can be stopped 
and edited ?aore easily. 

C. Saople Responses and Suggested Scoring 

*> 

I. Inadequate but broad scope saiiple response 



II- 



fi IBS trips as conpared to ajovies: are cheaper to produce, require 
less technical skill and planning to produce, are adaptable to a 
wider variety of topics and situations, are aore easily aiaintained 
and shipped, are less econonrical to store and retrieve for distri- 
bution, require fewer special skills to use 



This response received a "3" on the scope score (priniarily for the state- 
ments about adaptability and retrieval), but received only four points 
of the five required for a rating of "adequate" ("8" cn 0-10.s<ale), 
siiice^no meiytifln was nade of the important differences in inipact. 

Adequate but liaised stape sample response 



1 - 



niovies as conwared to filmstrips are snore expensive to produce, 
distribute, and use but arfusually much more exciting and moti- 
vating for students especially when action is eaphasized 



This response was judged "adequate" ("10" on 0-10 scale) since feasibility 
and impact were inentioned and differences for producers, distributors, and 
users were included; however, it received a "0" on the scope scale since • 
'this response did not demonstrate any knowledge other than the mimrauni 
required. 
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qOESTIOH 6: USER SATISFAmON 

A prograased text has been developed to teach hlg^-sdiool students the 
rudiaents of statistical inference. Indicate three questions a developer 
flight want to anhfer in a tryout about user (i.e.s student) satisfaction. 

A. Response Checklist 

interest in topic . 

abtivation for further study or use 



perceived worth of content 

perceived clarity, ease of use of text » 

perceived ability to achieve objectives 

attitudes toward mode of Instruction 

recoesnendations for lisproyement 

3- Suggested Criteria and Methods for Scorlrfg 

I. , Rating of ""adequate" or "inadequate-" 

in order to receive an "adequate" rating, the subject fwst receive 
a total of three checks; a maxintuni of two chaci^ inay be counted 
for any one category. 

II- Degree of adequacy score— score one point for each of the checks 

counted toward the total of three required for an "adequate" rating - 
(Convert to score on 0-10 scale.} 

Ill* Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant know- 
ledge brought to bear on the question on a scale of 0-4 (froai 
very limited to very broad). You might want to use the guide-- 
lines suggested under questlOT #1 . 

b) .exanipTes of possible iaiaginative responses indicating application 
' "of a broad range of relevant knowledge 

Mould you recommend the text to a friend, are you happy with the 
pacing, was the format a help or hinderance in your learning, 
could you see how the material could be relevant to your daily 
life? 
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SaHiple Responses and Suggested Scoring 
I. Inadequate sanple response 



a) can they solve statistical problems 

b) can they state estisiates trith the appropriate probability 
statenient 

c) do they %fapt to know core about statistical inference 



Only one of the conp onents of this response (c) is a "user satisfaction* 
question^ the others are questions about effectiveness. The raspGnse 
receives only one of the three points required (^3"* on the 0-10 scale) 
and receives a on scope. 

II. Adequate and broad sa}pe sasple response 



a) was the text organized in a logical and interesting vay 

b) were the exasples provided helpful in understanding the ciaterial 

c) did the text relate the sti>Ject matter adequately to possible 
ur^s in the subject's life 



This response covers **clar1ty/ease of use" and ""perceived worth of 
content" and is rated adequate ClO" on 0-10 scale). It also receives 
a "3" on scope prirarily for "c." 
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QUESTIOH 7: REVIEW VS. FiaO UST 

♦ 

What infonnaticn could you obtain from a consultant review of a product that 
you wuld not expect to obtain in a field test of the product? 

A* Response Checklist ^ 

content validity 

appropriateness/importarice of goals 

edi^rial axraents 

organization/sequencing of content 

possible additional resources 

as edia cc3e2patibili1y vrith content 

c omparison to similar snaterial on oarket/iHarketabiliJy 

theoretical or research base 

accuracy of content 

B. Suggested Criteria and Methods for Scoring 

I. Rating of ''adequate" or "inadequate." 

In order to receive an "'adequate" rating, the subject inust have 
any two of the lines checked. 

II. Degree of adequacy score—score one point for each line checked— 
maxiinum score is "two." (Convert to score on 0-10 scale.) 

III. Scop^. (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (from very limited 
to very creative). You might want to use the guidelines suggested 
under question II . 

b) exain>les of possible imaginative responses indicating application 
of a broad range of relevant knowledge 

Relationship of product and methodology to research, underlying 
theoretical assumptions, potential markets, findings of related 
field tests. 
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C- Sariple Responses and Suggested Scoring 
I. Inadequate sasple response 



review is based on logic and reasoning ratter than on observation 
review is based on the reviewer's unique background and expertise 

This very general response indicating the difference between review and 
field test does not really specify differences in infonnation likely to 
be obtained, and so would receive no points for adequacy ("0" on 0-10 
scale) or scope. 

II. Adequate but liaited scope sasiple response 



a) whether the content is accurate or not 

b) whether the important points are being stressed 

"This response addresses accuracy (a) and content validity (b) so would 
be judged adequate ("10'' on a 0-10 scale), but would be scored "0" for 
scope. * 

III* Adequate and broad scope response 

_a) adequacy and correctness of content 
fa) suggestions for alternate y^^ys of presenting material 

c) critical competitors 

d) liklihood of adoption by target audience 

e) appropriateness of cost 

f) other appropriate audiences 

g) adequacy of objectives 

This response is adequate ("10" on 0-10 scale) and would receive a "4" 
for scope primarily for b, c, d, e, and f* 
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QUES7I0H 8: QUESTIOHS FOR REVIBfSS 

« 

A large oil coopany has asked your R&D agency to develop a aethod of getting 
Information about the uses and processing of petrol etn and the present 
research taking place In the industry to upper elementao' schools. A decision 
was nsade to develop a self-contained presentation In the form of a l&m, 45 
fSinute colored fila. The film, shot on location, explains how petroleum Is 
refined, why new nethods of refinerent are needed and what is being done to 
find new asethods. In addition to the filn, a short booklet for cadi student 
has been developed which shows. In a cartoon sequence, the many uses petroleun 
has In our society. ^ • 

Although you have been woricing with the content expert provided by the cosspany, 
no one else outside your agency has seen your product. Below are'ttree people 
who have agreed to review your product. For each reviewer, list at least three 
tjuestlons you would ask them aboujt your product. 

1 . A 5th jrade teacher 

2- A sixth grade student 

3. The Oil Company Representative who initially asked you to produce 
the product- 

A. Response Checklist 

I- 5th Grade Teacher 

a) instructional effectiveness (e.g., will students understand 

it, what will they get out of it, is language level appro- 
priate) 

b) user satisfaction (e.g.,:will students enjoy it, is it too 

long) 

c) practicality of use {e.g., would you use it, can you run 

film, can you incorporate it into your class) 

d) recornmendations for revisions 

if. 6th Grade Student / 

a) instructional effectiveness (e.g., irfiat do you think you 

learned, knowledge questions about use/processing of oil, 
language level appropriateness) 

b) user satisfaction (^.g. , did you like it, would you rather 

^ , learn some other way, would you like to find out laore) 

c) practicality of use (e.g., do you need additional material. 

to use it) 

d) recooinendations for revisions 
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ill. Oil cccpany representative 

a) instructional effectiveness (e,g. , do you think the students 

will get vhat you mni^fJti^ to out of this) 

b) user satisfaction (e.g-, does it say iwiat you want it to) 

c) practicality of use (e.g,, can you afford it in its present 

fora, can you get schools to use it) 

d) recontnendations for revisions 

B. Suggested Criteria and Methods for Scoring 

I. Rating of "adequate" or "inadequate-" 

In order to receive an "adequate" rating, the subject nuist receive 
three checks for each of the three people to be questioned- For 
each person, two diecks fran any category cay be counted toward 
the required three - 

II- Degree of adequacy score~scx)re one point for each of the checks 
counted toward the total of nine required for an "adequate" rating- 
(Convert to score on 0-10 scale-) 

ill. Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant know- 
ledge brought to bear on the question on a scale of 0-4 (from 
very limited to very broad). You might want to use the guide-^ 
lines suggested under question #1. 

b) examples of possible imaginative responses indicating application 
of a broad range of relevant knowledge 

Promotability, what kind of evidence will convince you that the 
product is good, what are your reactions to the teacher and 
student co.Tinents, did you regard the film as propaganda, what 
would you like to leanj^-^'""^ 
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Sas^le Responses and Suggested Scoring 
I* Inadequate but broad scope response 



Teacher 

a) can you integrate tiiis film into your curriculum 

b) do stodents talk about the film aaong themselves 

c) do the students seen able to make the transition fran the filia 
to reality 

Sfaident 

d) did you enjoy the film 

e) explain why new met?iods are used to refine petroleum 

f) what would you like to learn about petroleum that was not 
covered In the film or booklet 

Oil Company Representative 

g) is the film accurate 

h) what kind of general impression does the film create in ele- 
mentary school children 

i) is this product what your company had in mind 



The parts of the response dealing with the student are 

adequate and broad in scope, they cover effectiyli|gfs (c and e), user 
satisfaction (b and d), feasibility (a),- and reconrfendations (f) and 
would rate a "3 or 4" on scope primarily for b, c, and f. However, only 
"i" is an acceptable question for the oil company representative, since 
"g" would be .known already from the consultation with the oil company's 
content expert and "h" should be asked of teachers op students. The^^ 
overall response, then, would receive 7 of the required 9 points ("8 
on the 0-10 scale) and would receive a "3" or "4" for scope* 
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QUESTIO« 9: OIFnCULTIES IN USE 

If the subjects you have selected for the tiyoyt of your product have diffiojHy 
in using the product effectively, what factors other than the product itself 
could have caused the difficulty? 

A.- Response Checklist 

users (e.g., lack of motivation, inadequate ability) 

packaging (e.g., rough, unattractive) 

setting/facili ties/time allotted 

interfacing (e.g., not in same context or sequence as, will be used) 

«. 

inadequate directiops/insufficient background infonnation 

evaluation procedures 

B- Suggested Criteria and Methods for Scoring 

I. Rating of "adequate" or "inadequate." 

In order io receive an "adequate" rating, the subject must receive 
three checks ~a maximum of two may be counted for any category. 

II. Degree of adequacy score— score one point for each of the' checks 

counted toward the total of three required for an "adequate" rating. 
(Convert to score on 0-10 scale.) 

III. Scope (of applied relevant knowledge) s^core 

a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (from very limited 
to very broad). You might want to use the guidelines suggested under 
question #1. 

b) examples of possible imaginative responses indicating application of 
a broad range of relevant knowledge 

Negative attitude of conductor of tryout, inappropriate evaluation 
criteria, lack of safeguards to prevent improper use, group Inter- 
action problems, resistance to trying something new. 
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Sample Responses and Suggested Scoring 
I. Inadequate sample response 



a) using the wrong medium to provide the instructions (e.g., 
written instructions to elefsentary school children 

b) giving the wrong directions 

c) poor administration of tryout 



This response describes three factors related to directions and in- 
structions and so only receives two points of the three required for 
a rating of adequate ("7" on 0-10 scale)- It might receive a "3" on 
scope for "a." 

II. Adequate and creative sample response 



conditions of test : a) teacher enthusiasm, b) teacher preparation, 

c) distractions, d) time alterations, 
e) variations in instructions 

f ) subject characteristics 

g) subject and test condition interactions 



This response covers users, setting > and direction factors jand would be 
judged adequate ("10" on 0-10 scale). It also would receive a "3" for 
scope primarily for a and g. 
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QUESTION 10: STEPS lU PRODUCiriG A FILHSTRIP 

Outline the sequence of major steps (at least five but not coore than ten) that 
are inost important in the production of a sound filmstrip between (a) the 
definition of the instructional objenives and (b) the collection of the first 
validation (pilot test) data. 

A. Response Checklist 

I. Outline 

treatment (i.e., rough outline of what should be in film 

sequencing) 

II. First version 

story board 

rough script 



III. Preparation for Evaluation 

development of pilot test instrument, selection of target 

group for evaluation 

IV. Preliminary production 

(arrange for) production of art work 

(arrange for) review of plan and art work 



V. Final production 

production/integration of slide sequence and audio tape 

Suggested Criteria and Methods for Scoring 

I. Rating of ''adequate" or "inadequate." 

In order to receive an "adequate" rating, the subject must receive 
at least five checks Including at least one for each category. 

II. Degree of aderquacy score—score one point for each category which 
has a check. (Convert to score on 0-10 scale.) 

III. Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (from very limited 
0 to very broad). You might want to use the guidelines suggested • 
under question #1. 



272 

^ ' n-24 

ERIC • ° ^ 



b) exainples of possible Imaginative responses indicating application 
of a broad range of relevant knowledge 

Relate instructional steps to objectives, synchronize the audio 
and the visuals, determine the constraints on film, consults with 
specialists. 

C- Sample Responses and Suggested Scoring 

'I. Inadequate sample response 



a) select site for filming 

b) prepare personnel at site 

c) make the fllmstrip 

d) try it out and revise as necessary 

e) pilot test it 



This response is too general to be an "adequate description of the es- 
sential steps in the production of a fllmstrip; "make the Tilmstrip" 
is covered in ojie step. This response would probably receive no points 
for adequacy unless "c" was counted as one of the steps in which case it 
would receive 1 point ("2" on 0-10 scale). It would pi^obabiy receive a 
"0" or "1" for scope. 

"II. Adequate sample response 



a) prepare criterion items 

b) prepare a product design 

c) draft a storyboard and script for audio 

d) review by subject matter expert 

e) create original set work required 

f) audition for soundtrack 

. g) create master copy of video and audio; synchronize and release 



All the major steps are^ covered in this response and so would be rated 
adequate ("10" on the 0-10 scale); it should be noted that a response 
doesn't have to include the "shop" vocabulary (e.g., "storyboard") 
in order to be judged adequate. .This response would probably receive a 
"3" for scope primarily because of a and d. 
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^' Appendix 
Scoring Manual ^or I^&E^piaulation basics 
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SCORIKS HAKUAL FOR DDiE SIMULATION TASKS 

Although the sinsulation t^sks will probably be more difficult and tiine 
consming to score than the knowledge test, we have tried to provide suggestions 
as to criteria and ssethods of scoring tiiese tasks sinilar to those we presented 
for the knowledge test. Vou pay want to develop youirpwn criteria^rrf/or 
scoring system to reflect the task and perfonBanC^ievel requirements of the 
position or Job iJiat the test taker laight be expected^ perform* 

As with the scoring guidelines for the knowledge te^t, we have suggested 
for each sinxilation task: 1) a checktist for respMseslor categories of responses, 
2) criteria and methods for scoring the adequa<gr of fresoonses, and 3) examples 
of responses that decnonstrate application of a^broSrf^5<ipe of relevant knowledge 
ttf, the questions as a guide to dete^ining a "scope" sco^e* In addition we have 
provided examples of how selertfed protocols frora tiie preliminary field test might 
be scored using our guidelines. 

The ^'response checklist" for each -question li54s those components of a 
response thatswe feel are important enough to be counted toward a rating of 
adequate. The "suggested criteria ^nd methods for scoring" indicates irfiich 
and how many of* the component^ listed in tte checklist we feel are required for 
a subject's response to be Judged "adequate." Tnat section also presents a 
method for deriving a "fraction score" for degree of adequacy • Although you may 
want to rely solely on a rating of "adequate" or "inadequate" for each response, 
we feel that a "fraction score" indicating how nearly a^response approaches ^ 
adequacy would also provide useful feedback to subjects- Suggestions for deriving 
a third score—t^ie scope (of relevant knowledge applied to a question)— are also 
provided in this section. Whereas the other two scores indicate how a response 
compares tff minimum requirements for adequacy, the "spope score" is independent 
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of these requirecsents and reflects the "breadth" or diversity of relevant 
knowledge applied to the question. For each question, s(xoe responses that night 
be judged to dsranstrate Jhe application of a broad rarige of relevant laiowl edge 
are suggested- A scoririg scale is presented under the "sc61>e score" section 
for question #1 as a guide for assigning the "scope score." At t!*is point, 
we don't feel that standards or requirecjents for the score can be pre- 
detemined so we only present suggested guidelines for what will be a very sub- 
' jective judgaent on your part- Despite this subjectivity, we feel that the 
scope of relevant faiowledge applied is an iasportant aspect of competence and 
should be included in assessinent- 

- Although you might want to develop your owi criteria and/or scoring 
systea, preliiainai^ field testing indicates that our suggKited scoring csethods 
have some validity! total scores on Sliailation tasks for adequacy of response 
and scope of response were only moderately related to each other, but they each 
.discriini;»at^betifeen students and professionals and were related to total scorps 
for adequacy and scope 'on the knowledge tests. 

Although, you would probably want to refer to the scoring manual yrfidle 
'scoring a subject's response to remind you of criteria arid methods, ft would be 
' .nwch more efficient and econociieal to use a-separate-suawapy- sheet iWk? ^^JP^^^^. 



the next page) to record the scores, since your evaluation of many responses 
can then be recorded on one page- iVis also suggested that all .subjects' 
responses "on one question are scored before proceeding to another question- 
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TASK 1: PREPm^Z GUIOaiKES FOR O^VaOPSlS OF THE LIERARIAfrS KAMEAl 



This task calls for the preparation of a cemo providing guidelines for 
the developers of Section^ of the Librarian's Banual for tii3~parent training 
session on the Feely Bag toy. You will have 1 1/4 hours to coenplete it- It 
should be no longer than the equivalent of three (3) double-spaced typed 
pages C900 words). You have been provided with several dcorsents (e.g*, Feely 
Bag toy^description and Section 1 of the Librarian's* Hanual —both attached to 
this oecso, a description of the Toy Lending Library, a description of your 
role as a member of the develofaent teaia) which should provide yo^j with the 
infonnation you will need to accomplish this task. You are free to re-read, 
refer to, and use these documents in any way during this part and later parts 
of the sinulation. However, please do mt refer to doqiroents included in 
later tasks. 

Remecjber, you are not responsible for actually developing the Librarian's 
Manual. Rather, you need to extract the relevant information from the documents 
provided and organize it in a memo so that the development team will know for 
whan the manual is to be written, in what form it is to be written, what it is 
supposed to accomplish, etc. Your memo should also outline the content of 
Section 2 of the manual, i.e., the steps the Librarian should follow in the 
parent training session (e.g., introducing parents, leading role-playing, etc.) 

A. Response Checkl ist 

(Check those responses indicated below that are included by the 
subject. Leave a response line blank if, by your judgment, the subject's 
response is not sufficiently similar to or does not meet the conditions 
specified in the designated response*) 
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The e1en)ents of the description of the manual (i.e., its speci- 
fications) are consistent with the inforcation in iiie provided 
documents. 

^ a) purpose— to enable the Librarian to plan and conduct the 

parent training session 

e.g. 5 to provide 'all the information needed for the 
leader of a training session; to familiarize 
the Jeader with the toy and its purposes and 
to help the leader with techniques and aethods 
for similarly familiarizing parents 

b) objectives 

1. are stated in behavioral teres (i.e., specify what 

the user should be able to do} 

2. Librarian should be able to: ^ • 

' plan the session 

introduce the toy and its purposes 

organize ^e role-playing session 

^ review the' Responsive Program Principles 

answer questions asked by the parents. 



c) fomiat 

1. instructions to Librarian should be double-spaced, 

numbered, and listed in order as they are to be 
used 

2. sections to be read aloud to parents should be trip! 

spaced and boxed 

d) ^jser group Xi-^.e->Jjj3rarians}--Tdurts who are part of the 
community of the target parents 

e^g., adults who live in the same community as the 

parents; '^neighbors^ of the target parents who , 
have about the same incomes as those parents 



_e) user activities— Librarian will read the Manual before 
conducting the training session and will refer to it 
during the session 

j) constraints— Appropriate difficulty 

e.g., no technical terms, not above an 8th grade 
reading level 



il- The outline of the content of the Hanual includes instructions 
to the Librarian for his/her oost iaportant tasks. 



a) 



1- operating and disojssing the filasta*ip 

2- "leading role-playir^ 

_3. answering questions froQ parents 



b) 



J. planning the session 

_2. introducing parents to. each other 

3- introducing the toy and its purposes 

_4. specifying *fhat parents need to do 

_5. discussing the Responsive principle 



III. The guidelines are well-written. 

Rate each of the following on a 4-point scale where: 

1 - very poor/inadequate 

2 - weak, of marginal adequacy 

3 - good/quite adequate 

4 - excellent 

a) organization: logical sequence, smooth transitions 

b) style: clarity, fluidity 



c) language: grammar, spelling, appropriate use of words 



(Check each line a-c for which the rating was "3" or "4". If only 
one rating is "-Z"^ it also receives a check; if more than one 
rating was "2", give only one of than a check.) 
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IV. Directions/instructions were follwed. 



a) provides enough detail in specifications/guidelines so 

that developers could develop tJie actual Librarian's 
Hanual 

b) presents guidelines for developers rather than attenpting 

to develop the Kamial 

c) is of appropriate lengtlrtl. If •V'ncrlonger than Uie equiva- — 

lent of three double-spaced typed pages~900 words 

8. Suggested Criteria and Methods for Scoring 

(ITiese are only suggested criteria and methods; you isay want to 
develop your own.) 

I. Overall rating of "adequate" or "inadeijuate." 

In order to receive an "adequate" rating, the subject nwst receive: 
a) checks on all the lines for I, III, and IV except that only two 
objectives (Ib2) are required, b) checks on all three lines for Ila 
and on two lines for lib. 

II. Degree of adequacy score — score one fioint for each of the checks 

counted toward the 20 yequired for ah "adequate"* rating. (In order 
to obtain equivalent scores for each of the simulation exercises, 
you will probably want to convert the point scores into scores on 
a standard scale,<^.g., 4 of a possible 20 points on Task I would 
receive a score of "2" on a 0-10 scile as would 3 of a possible 15 
points on Task 3.) 

III. Scope (of applied relevant knowledge) score 

a) rate the subject's response as to the scope of relevant 
knowledge brought to bear on the question on a scale of 0-4 
(very limited to very broad). This score is independent of 
the adequacy score—a subject's response csuld be judged 
broad in scope even though It did not include all the required 
aspects. 'You might v/ant to use the following guidelines to 
help assign this scope score. , ^ 

0 - Included nothing other ttan responses required 

1 - included one of two responses which were not required and which 

you judged to be acceptable but not especially imaginative or 
Indicative of a broad scope of relevant knowledge 

2 - included a great number of responses irtiich were not required and 

which -you judged to be acceptable but not especially imaginative 
or indicative of a broad scope of relevant knowledge 

3 - Included one or two responses which were not required and may not 

even have been on the checklist which you judged to be acceptable 
and very Imaginative or indlcatiye of a broad scope of relevant 
knowledge 

4 - Included many responses which were not required and may noj even 

have been on t^ie checklist which you judged to be acceptable and 
very imaginative or Indicative of a broad scope of relevant 
knowledge 
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b) exaniples of possible "'imaginative" resfK)nses indicating 
application of a broad scope of relevant knowledge 

Instnictions to Librarian—debrief irg session with suggested 
questions for Librarian to ask parents; peer evaluation of 
role-playing. 

C. Sasple Responses and Suggested Scoring 

The following protocol, reproduced in its entirety (with cements 
keyed to nunibered parts of the response) should provide sose guidance as 
to possible criteria and scoring of Task #1. 



PROTOCOL " ^ 

TASK 1: Preparing Guidelines for Developers of the Librarian's Manual 
BackgrounJ Information 

Background information on the Parent/Child Toy Lending Library is 
included in the Librarian's Manual for Session 1, and parents will already 
be familiar with it. Therefore, detailed specifications of the theory and 
pedogogical orientation underlying the Lending Library is not included 
in the design for Session 2. Reference to this fact will be made at an 
appropriate point in Manual 2. (se^ fonnat section below)® 
Product Description 

— \ 

Manual 2 of the Parent/Chile Toy Lending Library is designed to serve 

as the guide for the Librarian who will be leading the parent training 

session explaining the educational use of the Feely Bag tdy, the second toy 

m the eight-week lending-library course^ Users of the product practically 

comprise two groups: 1) the "librarian" or other service personnel in 

charge of administering the Parent/Child Toy Lending Library; since agencies 

scheduled as target audiences (buyers/renters) of the Toy Le/iding Library 

, include public libraries, churches, community centers, hospitals, schools, 

day-care centers, or simply interested parents who organize to use tte ' . 

library, the Manual must be appropriate to ©the mfddle-level profess'ional 

(3) 

staff of all th^se agencies* Therefore, the language and instructions 
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oust be clear, direct, not couched in heavily technical or theoretical 
® 

terss. 2) The ultiniate users of the Parent/Child Toy Lending Library 
are the parents themselves. Potential users of this group include all 
families irfiose income is above the level qualifying them for Head Start 
but not high enough to^afford private nursery school. As this criterion ' 
defines 3/5 of the country's population, users will cover a broad spectruci 
of social, educational and professional levels. Instructions included in 
the HanUal inust be phrased in a language comprehensible to all groups; 

® 

it should reflect approximately an 8th grade reading comprehension level. 

The Hanual itself is to be used only by the librarian, primarily before 
the session (as orientation material) and also referred to during the session- 
It should provide a thorough and clear description of the Feely Bag toy, 
of the educational purposes of the toy, and of the gentatlon, practice, and 
role-playing in which the parents will participate. Objectives of Manual 
2 are: 

1- To familiarize the librarian (or other "trainer") with the 
Feely Bag toy and with how it is used- 

2- To enable the librarian rto expljain this toy and its use to the 

(D 

parent users during a formal training session- 
Activities which the librarian/course leader y#ill engage in while working 
with the manua.l are: reading the manual; working with tte. Feely Bag in 
order to become familiar with its characteristics; viewing the accompanying 
filmstrip; reading the accompanying instrifctions to parents. Other user 
activitie.s, sucfi as practice training sessions and reading additional source 
material, may be added at the user's discretion- 
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Production, user, and distribution constraints on Manual 2 are- 
it is to be printed on newsprint, thereby making it cheaper to users, 
And making later additional printings, as dieap as possible, it is to 
be spiral bound between staff cardboard covers; the printer can accomraotjate . 
no more than 500 of these on a single run; the light paper construction 
is necessitated partly by shipping costs - all materials for each session 
of the 8-week Parent/Child Toy tending Librai^L.course-wi44-be-del4vered 
parcel post (U.S. mail). Users require direct, clear language unencunbered » 
by technical jargon or cdraplicated rationales. Manual 2 is to have the 
following characteristics: 

1. No more than 20 pages in length (printed both sides). 

2. Printed on newsprint pages, 8 1/2 x 11", in double-spaced 
gothic script with 1 1/2" margins on all sides- 

3. Bound between 2 glossy stiff cardboard covers: the front cover 
will include a) the title ("Librarian's Manual, Session 2: 
Feely Bag toy", b) a 3 color cartoon illustration of the toy, ar^l 
c) the developer's name and the words "Parent/Child Toy Lending 
Library". Manual 2 will have a tan ground with a blue, red 

and yellow illustration- 

4. First edition print 'of *500. 
The format of » Manual 2 is as follows: 

!• Introduction to librarian (2 pages, no more). 

a. ' An introductory blurt describing the Feely Bag toy (on 

page .4 -of Task 1 instruction). 

b. Brief discussion of its purpose and of the principle 



illustrateif by this toy. 
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2. Instructions on how to demonstrate the toy {1/2 page). 

3. Librarian's introduction to parents: 

a. Stateroent of the purpose of this introduction (1 paragraph) - 
something about- need to introduce this toy and how its. 
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purpose differs from that of the first session's toy. 
fa- Instructions to parents (to be read or referred to in 

discussion, at leader's discretion). Instructions to parents 
should be boxed, triple space, in italics. They should 
refer to the following major points and elaborate on these: 

1. Description of the game: "One set of shapes are placed in 
the' drawstring bag.,," (see pi4 of Task 1 instructions).^ 

2. The^ child should be asked once a day If he or she wants to 
play with the toy. 

3. The child may change the rules of the game at any time and 
the parent should tcquiesce. 

4. Parents should stop the gamejwhen the child loses Interest.® 
4- Instructions to distribute Feely Bags to each parent, to let 

them examine them a while. ' ' 

5. Interlude to show the filmstrip. 

6. Instructions on the purpose of the role-playing session (referring 
to analogous section in Manual 1 for greater det|n) and on how to 
initiate and coordinate the role-playing. Include sample protocols 
from pilot test as ^^P^^ role-playing (Interlude for role- 
playing). (3 pages)^ < 

7. Guidelines for debriefing stsslon (5-6 pages) Including: <^ 
a) A list of questions to ask parents (hypothetical questions 

to stimulate discusslofif: .e.g.., "what would you do If your 
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child was unable to guess any shapes the first time 
around? How could you make it easier to start?" "Suppose 
your older child wanted to play too - how could you make 
it fun for both of them, help the older child to teach 
the ycunger?", etc*) 
b) Samples of comments made by pilot users (e.g., "great 
fun", "hard to 4ise at first but very rewarding") to 
add human elemeht/enthusiasm/etc. A cheery closing note. 
8. ' "Any questions?" 



KEYED SCORING GUIDE TO PROTOCOL FOR TASK 1 



Number on 

response checklist 



Numbered part of 
subject's response 



la: purpose 



lb: objectives 



(D and (7) . 

taken together indicate that the purpose of the 
manual is to familiarize the librarian with the 
uses of the toy and with methods of teaching 
these uses to parents (so la receives a check). 

(8) 

is stated in behavioral terms (so Ibl is checked) 
but is limited to enabling the librarian to 
explain the toy and its uses. A developer of the 
manual should know that the manual needs to focus 
specifically, on enabling the 1 jbf'arian to or^ani 
i^ole-playing etc. - objectives which ar^.specifi 
in the document "Your role as a member of 
development team." The response "receiv^...0Tny 1 
of the required 2 checks for Ib2. 



a 



format 



user group 



user activities 



constraints 



essential instructions 
for libr&riar 



other instructions 
for librarian 



specifies that instructions to be read to 
parents; should be triple-spaced and boxed* 
(so Ic2 is checked). The entire "fomat" 
section of the response numbers and lists 
instructions and procedures for the librarian; 
assuming that this is a model for the manual 
developer to follow, Icl a.lsp receives a check. 

@ and © 

taken together imply that^^e librarians will 
be adult members of the^rget conimjnities 
(so Id receives a che6k). 

® • 



indicates pi^t the librarfan-Will read the 
manual l^efore the session .and will refer to 
it 'dujf>ing the session (so le receives a check). 

($, @ and @ 

specify a clear, non-technical, 8th grade 
reading level (so If receives a check). 

@ and (l 

indicate that Instructions for conducting role- 
playing and ansmritig questions are to be 
included. However, since there 1^ no specific 
cation that instructions for running and 
jdiscussing thetfilm are to be inclpded, the 
response receives only 2 of the required 3 
checks for Ila^ 



@f and @ 



^specify that instructions for t^^ibrafilan 
relevant to the Responsiv^e •Prjnclfle,. introducin 
the toy and Its purposes,. and su^sting things 
for parents to do with their children while playing 
with the toy- are Included (so lib receives the 
maximum scorable 2 che'cks). 

Style and language are rated^"4^ for excellence; 
the organization was judged to be generally good 
though somewhat redundant so v. .s rated "3". 
(So III received 3 checks^. ^ 
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Checks were given for all three categories 
involving the following of directions; the' 
response was judged to be detailed enough to 
enable development of the manual, but was- 
still an outline of what should be included 
in the manual rather than an attempt to . 
actually write the manual. 

A total of 18 of the 20 checks required for an "adequate" rating were 
received, so this response might be judged inadequate, but would receive 
a "9"' on the 0-10 degree of adequacy score. 

The response might receive a "4" for scope of applied relevant knowledge 
primarily for^the descriptions of: developer constraints and specifications 
(10) t rules parents should follow or explain to their children in using the 
toy (14), "debriefing" procedures for the librarian to follow (16), an^ the 
relationship' between sections 1 and^2 of the manual .(IJ. 



t*-.n t25i ycj are to: (1) iescribe what t?;e Eesearc^i and Evalyaticn 
Ic-J- for in selectir.g test sites. librariariS, ard parents, ar.d 
^2] Tist t*:e qjesticns (separately for litrariar-s, parents, and cb5er.?ers) 
f^at /Ci wcjld lifce to tiave ar.SKered in a trycut of trie Librarian's v=r.ual 

t*''at "ft *';i3»'e dsvelcped- 

T»-.e Hst of c,j5st1cr.s is the ncre inportant of t*:e tKO activities- 
It s*ieul j ta*'e rxj Icnger t^iarj t*:e eqiiivalenl of (2) typewritten dr^ble- 
sraced pages . 

^ne iescripticn for the Pesearc^i and Evaluation Tesn should be 
brief-- ere or t«o paragraphs - 

This task should take ro ~ore t*i3n 30 niirjtes. 
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1- I.£rScr1ptiort of v^tiat to IrcJ' for in selection of tr/out sites and 
Lt-.^^rs is c^n^ir^teT^t with inforraticii in provided docurtents. 

di) site — aC'proDriateness to intended use 

e.g., in cuirunity— schcol , library, church, union, etc. 

p) librarians^-ccT-unitv Ter.bers 

c) Darents—Jippronriate (as Jesi^nated) incone level, parents 

of 3-4 yaar olds 

II. Questions for tne librarians, parents, and observer are appropriate, 
useful, and co"^prehansive- 

(C^,eck the apDropriate category for each of the subject's ques- 
tions—a category ^ay, then, have rore than one check.) 

a) Librarian 

1. instructiontal effectiveness 

e.^., were instructions clear, did it qive you 
enough guidance, did it help you feel 
cornfortable and competent in runninq 
session 

263 



1. -i^^r S3t^ ' ^iC.tlirt 

t.'^., ulj v:j erroy ^s1^i-5 the -^iruBl i^'J run- 
•'li'nn :-^sr.lcn, w^ijld *^cj nJe to It 

3. tracts C2lHv of lise 

/ry refer to ttB ^3-'niial e^s^ly sfi^rlng tfce 

recDTsr/fstlcfiS for revlslcns 

instn^ctknal effectr*^aness 

€.a., was the Librarian a ^elp irj leamirirj tciir: to 
jse the tccv', can you nt^ ^ypl^in the noirC'Dses 
arjd use of t^.e tcy 

2. >s^r satisfaction 

£--5-, did V-Cu anjoy the session, Kculd voii core to 
5f:Dt':er one, ^'as the Librarian liriderstand- 
ir^g/ organized 

^3. practicality of use 

8-t3., ^A^as the sessicn nore trcyble than it was 
rorth 

4. reccn^^ndations for revisions 

c) Observer 

1. instructional effectiveness 

e-g. , did the parents leam how to use the toy, 
did they understand the Responsive 
principle, ivas the lanquax?e level aporo- 
oriate 

2^ user satisfaction 

e.q., did the Librarian/parents enjoy the ses- 
sion, did the parents resent the Libra- 
rian's being in charge, did they feel the 
session was v;orth;i?hile, did the parents 
use the toys with their children following 
the session 

3. practicality of me 

e.g., were there any equipr^nt problens, could 
t^a Librarian use the x-nanual s'^oothlv 

4. reccrrendations for revisions 
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III. "^v circCtlcr^/irstr^ctitriS ^^r^ follD^e^. 

^) -^frstnr.: lasted sef3r^iM-!v -^ur lUraHsn, rs^^^tts, ard 

? ^'I'^-irHit^ ler.Qtii {i.e., 1-2 parasrsr^is for site 5f;d s:-l- 
;ect5; f 0 Itrjar t^arj equivalent of do^ible-s^aced tvreiS 
-ACr^S—fDr questicns) 

^:.';'r-:t'^: Criteria Tfet^ods for ScDrir.g. 

I. C*i-r2:11 ratir:g of "atieqii^te*^ cr "?r.adequ2te-*' 

1. £rcer to rcvei^e '^adequate" r^tirg, the subject r^st -^^ei^^e: 
j; ejects cfi all lir.es for I and il!, and fc) a total of three c^ecfcs 
f:r eart of lis, Ufa, and lie ^^here at least cne atid no r-.ore than 
tvD 0^ tr:Dr-e cnecJs Is for ^'effectiveness.*' 

II. Degree of adequacy score— score one point for each of the checks 
counted toward the 14 required for an "sdeqjate** rating. (Concert 
point score into equivalent on C-10 scale.) 

III. Sccpe (of applied relevant knowl^ge) score 

a) rate the subiect's response as to the scope relevant 
knowledge brought to bear on the question on a scale of 
0-4 (very limited to very broad). This score is independent 
of the adequacy score—a subject's response could be judged 
broad in scope" even though it did not include all the required 
aspects. You night want to use the guidelines suggested 
under Task #1 to help assign this <^ropB score- 
fa) exarq^les of possible "inaginative** responses indicating 
application of a broad range of relevant knowledge- 
Consideration of feasibility (cost, distance) of sites, 
consideration of variety of sites/parents/librarians, 
specification of follow-up questions for parents after trying 
toy with child. 

Sanple Responses and Suggested Scoring. 

The following protocol * judged to be Inadequate, reproduced in itc 
entirety (with ca-nents about scoring keyed to nunbered parts of the 
response) should provide so^e guidance as to possible criteria and 
scoring of task «2- 
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PROTOCOL 

TASK 2: P££P^.gl!i3 FCH 1¥1 ViWJT CF THE RESOITXE KIT 
CcnditiofjS 

Tfce ptpAatitn needed to tr>'OJt this should contain no less than i^o 
librarians. Eacfi librarian should have abDut six parents as students. ^ 
Individuals selected to act as librarians should be adult high school graduates. 

The test site should ccnsist of a large rcan with tables lar^e enough to 
conduct role playirg exercises. Electrical power outlet required. Lights 
with cn/off switch required as tryout will probably have to be conducted in 

■ © 

the evening. 
Questions 

The following inforration should be solicited: 
a. Librarian 

1. Were you successful? 

2. Did you enjoy the activity? 

3. k^Duld you conduct ^ similar session? 

4. How long did it take you to prepare? 

5. What additional aids could you use to niake your job as librarian 
easier? ^ 

(D 

6. Specify problens you had in using the sanual. 
b. Parents 

1. Can you use the Feely Bag toy after going through the session? 
2- iJhich activity helped you n:ost? 

a. Introduction 

b. Deronstration 

c. Role-playing 

d. Individual help after role-playing 

9 ) > 



® 

i. k'cjid /DJ tt;rTV^ and use this toy wit^t /ojr c^ild? 
c- Observer 

1. K3S &ie librarian Eble to lead tte parents through tfie learnirg 
exferlerces with ease? 

2. ^ote points of confusion cbsen^ed dzirir.g the presentation by 
the librarian and parents. 

3. Evaluate the attltude^f the parents at eath stage of the session 
cn ore to five scale. 



KEYED SCCRIKG GUIDE TO PROTOCOL FOR TASK 2 



TiurDer cn fJunbered part of 

response checl^llst subject's response 



la: (D 



describes physical characteristics of the 
rooni relevant to its suitability for 
conducting role-playing, etc., but since 
itriakes no 'mention of the more fundar.ental 
criteria for test-sites involving location 
(e.g., "in surroundings faniliar to users, 
easy to get to**) no check for ''site*' would 
be given. 

lb: ® 

specifies only that librarians should be 
adult high school graduates. Since it does 
not specify that they should be members of 
the conmunity who have about the same incane 
level as users, no check for "librarians" 
would be given. 

Ic: To receive a check for "parents", sofne 

mention should have been made of their 
inccxne level and the age of their children. 
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'iliriered part of 
subject's response 



® 

Ti-e Questicns fcr the librarian cover -effectivenp" 
(1.6). -user satisfaction" {2.3). "practicality of 
us^ (4). arid "recorcendations" (5). but sons c. the 
questions (e.g.. 1 and 2) are very general and of 
questicnable use to the field tester- :oj night 
decide t^ey are adequate, but you would unaoubtedly 
give little credit for "scope". The following 
questicns fro:a another protocol would provide rore 
useful inforratior.: 

Here you able to follow the sequence of the Manual 

if not. "what happened that caused you to change 
your sequence? 

Mere you able to give quick, confident answers 
to Questions asked? 

If not, what were tte questions you were not prepared 

could you follow the I'^nual conveniently during the 
session or did you have trouble finding your place? 
If you had trouble, where? , 
Would you want to use a similar cianual m .uture 

sessions? ^ , -„ 

What changes, if any. would you like to see m the 

nanual? 

© 

The questions for the parents involve "ejfff ^^eness'' 
(1 and 2) and "user satisraction (3) and so could oe 
iudqed "a. quate"; however, again they are very 
genlral ana'the yes/no answers lijffly.to be obtained 
Sould be of liniited usefulness. The roll o-wing 
questions from another protocol would provide more 
useful information: 
At what points, if any. was the librarian confused 

Sh'af— red questions, if any ^^o jou still have 
about to play the game or what your child can 

learn from the game? ^ 4- „f fho cP«:«;ion 

Did any problems arise in the conduct of the session 

that lessened your enjoyment of it? 

What were they? 
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Ttelser on 

response rtecklist 



^ioriered part of 
subject's response 



lie: d) 

Tne questions for the observer involve "effectiveness* 
(2), "user satisfaction" (3), and "practicality" (1) 
and «oiild probably yield valuable Inforoation, so 
Kculd pnsbably be judged "adequate: (i.e.? receive 
checks), ^^any other questions about procedures 
during the session, parent participation, and 
outcomes for the parents- would be useful - 

III: The directions concerning length and separate 

questions for librarians, parents, and observer 
were followed, so Ilia and Illb would receive 
checks. 

Since this response did not receive any checks for i and night not have 
received all the required checks for Ila or fib (because of the generality of 
the questions), it would be scored fron 5-11 on "degree of adequacy" and would 
be judged "inadequate." Very little imagination or breadth of relevant applied 
knowledge was denonstrated, so it might receive a "1" on the scope score for the 
description of the requirements of the test site room (2) and the specification 
of a parent attitude scale for the observer to fill out (5). 
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T^ere are two activities invDlved In this task. Listed in order of 
^•^portarxe they are: 

1. Reccnr.ending revlsicns for the Librarian's Manual. {This product 
should be no longer than tJie equivalent of three (3) double-spaced 
typed pages.) 

2. Critiquing the tryout. (This product should be no longer than 
the equivalent of two (2) double-spaced typed pages.) 

You have 1 hour to canplete this task. 



ResDonse Checklist 

I. Suggested revisions of f'*anual address problems indicated in the 
inforratlon provided {in the Tryout Report). 

^a) role-olaylnq 

e.g., additional practice, filni, nodels, r.ore back- 
ground infonpatlon for Librarian 

^b) Responsive pHnciple 

e.g., more discussion application, more chance for 
feedback and Input from parents 

^c) Librarian fo^lov/lng nanual 

e.g., r.ore specific Instructions to follow manual 
step-by-step, more opportunity for Librarian 
to paraphrase 

^d) ansv;er1ng questions from narents 

e.q., more examples of likely questions and how to 
handle them 

^e) Librarian's experience and expertise 

e.g., manual made more flexible to varying degrees 
of Librarian skill and knowledge, optional 
background information 

f) need for Introductory material about proqram and toy If 

not used In senuence after a previous session 
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li. Ctardcterutlcs identified as strengths -weaknesses of tryout 
ar^ reasonable- ar:d comprehensive. 

{Kark a check for each strength or weakness included which 
you judge to te reasonable and i-^portant. Each line, then, can 
have several checks-) 

a) strengths 

e.9-. 

variety of sites/librarians/parents 

Librarian and parents knew each other 

detailed descriptions of sites and users in report 

observed parents as well as Librarians 

observer gave irpressions and suggestions as we^l as 
record of events 

oarents and Librarians inter¥4ev;ed 
follcw-up on parents* actual use of toys 
quantified results of interviews with parents 
observer adopted procedure to situation at hand 

^b) weaknesses 

e.g.. 

Librarians not given manual soon enough 

no objective assessment of how well objective met 

no indication of how parents recruited 

observer imposed self into instruction in second 
test site 

tested session out of context {no preceding session) 

observer didn't probe after initial questions (e.g.. 
Librarian wants longer film, but what additional con- 
tent is wanted?) 

parent interviev/s not done anonymously, so may get 
halo effect 

Librarian recruited parents—may be biased sample 

confusion of what being pilot tested (i.e., manual or 
film) 

no evidence of interview v/ith second Librarian 

same sites used as for testing prototype lesson- 
limited sample 

297 

ERiC " 



Suqqested modifications for tryout address weakness in tryout 
• pSure or inadequacies in tryout outconies. 

(Hark a check for each nodification you judge is likely to 
remedy a weakness in the tryout.) 

a) selection of test site(s). librarians' parents 
— rSperirnif^n teaching 

or working with parents 

makina arrangement for the tryout 
b) ^^^S^riiatlrials to librarians earlier 

c) directions which should be given to the librarian(s) before 

the session ^pQuence in manual should be followed, 

d) specific directions and questions for the evaluator observing 

T.Vlls:^er Should not impose -i;;^--i?-.;r""" 
fhould be conducted anonyifously. probe in follow up 

questions to parents. 
IV The directions/instructions were followed. 



Suggested Criteria and Methods for Scoring. 
I Overall rating of "adequate" or "Inadequate." 

,„ order to receive an "a-equate" rating -^^jS/f^J/fn"; ; 
?lannJl!j;2rere^^s%:a^eLf tlrlfof^he four categories for lU 
and d) checks on both lines of IV. 

• + for pach of t-he checks 
point score to equivalent on 0-10 scale.} 



III. Scope {of applied relevant knowledge) score 



a) rate the subject's response as to the scope of relevant knowledge 
brought to bear on the question on a scale of 0-4 (very limited 

to very broad). This score is independent of the adequacy score— 
d subject's response could be judged broad in scope even though 
it did not include all the required aspects. You might want to 
use the guidelines suggested under Task 1 to help assign this score 

b) examples of possible "imaginative" responses indicating a broad 
range of relevant knov/ledge. 

Manual should contain brief description of teaching and 
group leading skills and techniques, include report of 
specific questions and responses in interviev/; need 
group as well as individual "debriefing.-*' 

C. Sample Responses and Suggested Scoring 

The following protocol, reproduced in its entirety (with comments 

about scoring keyed to numbered parts of the response) should provide 

some guidance as to possible criteria and scoring of task r3. 



PROTOCOL 

TASK 3: RECOMMENDING REVISIONS FOR THE LIBRARIAN'S MANUAL 

1. Give Librarians the manual a week ahead of the training session, so 
they'll have time to work out difficulties and problems. 

2. Emphasize in the beginning of the manual that for those who are 
fairly inexperienced in leading this kind of activity, following the 
guide closely shou-ld be a big help until the librarian feels more 
confident and can branch out more spontaneously. 

3. Include in more detail an explanation of the Toy Lending Library and 
the Far West Laboratory in the manual, which if necessary can be 
read to the parents. 

/ 
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4. Include a section for feedback from the v/eek at home with the previous 
toy, right after the introduction. This is a good method of breaking 
the ice, getting parents to speak out informally, and encouraging 
them to try the toy v/ith their children. 

5. Include all the Responsive Program Principles in each manual. {It 
might also be a good idea to include a chart listing these principles,^ 
v/hich can be hung up for each session.) Place special emphasis on 

the one being discussed at that session and give several examples 
of this principle in section five. 

6. Give a more detailed explanatjaji of role-playing and include one , 
example of possible dialogue and movements v;hich might occur with 
that particular toy when a child and parent use it. The librarian 
and a selected parent can demonstrate exactly v/hat is in the manual 
for the first role-playing attempt. 

7. Include a list of the most commonly asked questions of parents and 
possible answers. 

8. Give each parent a Feely Bag Toy and a set of instructions when the 
librarian introduces the toy. Parents can examine the toy while the 
librarian is speaking about it and can read the rules 'bn how to play 
the game before they view the films trip. 

9. Make sure the manual is v/ell -coordinated with the films trip. If 

the manual calls for role-playing or discussion .of a Responsive Pi^ogram 
Principle make sure both are previously demonstrated or discussed in 
the films trip. 

10. Include a list of possible variations on tfie original Feely Bag Game. 
Include discussion and role-piaying of some of these variations as part 
of the manual's instructions, to the librarian. 

3 ;0 
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• r r-^j^ira fjr^ lifcrarian re»new t^ie Feely Bag Toy and It s 
^£^, 'et ^'er ccrdiict ^er review in t^e fom of nuesticns for the 
::j^^r-ts- >iU v;*?!! let ^er know if parents truly understand Kfiat 
♦-ass tee'-' de-- castrated and it will allow t^-ie librarian to clear up 

E'^a f^.e session cn a pleasant and opt^"''istic note ard tiave tne lib»"ar^an 
e":cu*^3ce i-arentt to use t*-^e toy at ^-lO-^-e- 

*i^.>t ci^e t^ie -^ajo^ s'^'engths? 
. "^^e choice of sites was good-- both being in low-inccre 
p^edo'^inantly ninority neighborhoods. 

2. >e location of each session was noma! to tt^e participants. 

3. Li&rsrians and parents we>^e inter/iewed by the obser/er. 
^. T^-ie Parents v;ere followed up by telephone. 

5. The lib'^arians were given help in advance in order to prepare 
for t^»e lesson. 

t. ^^e racial ca-^position of the sessions was representative of 
the neighborhoods, although this my have been accidental since 
no qyidelines were given to the librarians for recnjltnient. 

7. Both librarians were nembers of the cc^munitles. 

8. Both experienced and inexperienceo teachers were used. 
VJnat are the weaknesses? 

1. Each librarian should have been required to recmit at least 8 
parents, to provide for attenuation. 

2. One session was held at 10:00 a.n. when working parents would 

be excluded tron attending. (Such parents may be more represent- 
ative of those in the neighborhood.) 
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3, Tte unferillarn/ of ^-arents wit^i the Toy Lerdirg Library 
*s?3s f.Dt anticipated. 

4. Tte evaluator tecere ifivolved In cne sessicn, so that it carxot 
be CDnsldered ritual. 

z. Cne librarian did rot foil cm the naniial. 

6. Parents ware rot followed with regard to their irple-entation 

of thtf Resnonsive Prcgran Principles. 
IrstructiOfiS ^or future try outs 

1. Be sure that racial balance is representative of neighborhood. 

2. Be sure fathers are included if possible. 

3. Be sure that librarian is prepared to give orientation to 
new parents. 

4. Be sure that librarian understands that he/she should follow 
the --anual with regard to activities. 

5. Be sure ttiat evaluator does not beccr.e a teacher. 

6. Haye evaluator ask questions regarding iriplenentation of the 
Responsive Prograrr^ Principles in the follow-up telephone call: . 
For exanple, 

a. Did your child nake up his/her own rules? 

b. Did your child not want to play any tine? If so, what did 
you do? 

c. Did your child want to play with the toy without using 
rules? If so, what did you do? 




t'IfED ItL^r.'i 3'JII£ TO PPDTOCDL FC5 T^5t: 3 



I: fJlany of the prolilBr^ Indicated in the tryoiit report 

are addressed in this protocol: role-playinq (a) 
by revisicT} ''o*', the respMsive principle {bj by ^S"^ 
fij I lowing the fnanual (c) by ^Z'', ansKeriiig questicns 
{d) by "7*', and imrcductcry luaterial (f) by ""3^. 
(The response receives 5 checks for "revisicns of 
-manual" though cnly 4 are counted toward 1±ie total 
score.) In addition, the subject has incljided 
several other revisions Khich show imagination and a 
bread understendinq of potential problems and possible 
solutions. 

I!a: The listing of strengths of the tryout covers aspects 

of: the sites used (l arid 2), the personnel used (6 
and 7), the level of experience and preparation of 
the librarians (5 and 8), and the scope of information- 
seeking by the observer {3 and 4). The character- 
istics listed cover a fairly bread range of aspects of 
tryout design and implei^entation and could be judged to 
be inportant strengths of the tryout- "Strengths'" 
{I la) v?ould receive i!:ore tftan the 3 checks required to 
be judged ^adequate." 

lib: The listing of weaknesses of the tryout covers aspects 

of: size and bias of the sanple (1 and 2), lack of 
appropriate context for the field test (3), ncn- 
objectivity of the observer (4), non-use of the manual 

(5) , and lack of appropriate follow-up of parents 

(6) - ''Weaknesses*' (lib) would receive Dore than the 3 
checks required to be judged ''adequate.'* 

HI: The subject^s reconnendations for a second "tryout 

address three of the four areas specified in jthe task 
statement (i.e., site and personnel, preparation of 
librarian, directions to observer) and address nost 
of the aspects of the tryout previously identified as 
weaknesses in lib. '^Modifications" (III) receives the 
three checks required to be "adequate." 
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resccrse dnezkMst ^ 

I J. ' The length restricticrss were not exceeded, 

as tte two required checks for follcwing dfrecticriS 
were received- 

Tms response received all 15 of tiie checks recuired for a rating of 
"adequate." It. additicn, there were cany evidences of application of broad 
kf:owled5e and understanding rele/ant to tryouts and rensions {e.g., suggested 
revisions of the oanual U, fS, 19, flO, and ill; the weakness involving serple 
bias, #2; and the specific follow-up questions for tte observer to ask ibe 
parents, #6)- This response, then, would be judged "adequate" and could receive 
a "4" scope score. 
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vV-." , . V iTfficjIt ticit ',ir.c- yt^n -^ill t€ given rnly 1 tf.ur to vCTi ,t. 

r.l'»*-7^,'i' ^ -it*. nr*r.- plavir^q totfi t>e parent's ar.d t»:e child's r£.1e£, in 
. -.- -i.r. tr. .^cicfi 3 -ar-^nf is teachir^ a "child" hc.« to play «ith tte Feel/ 

i r,., l*.t^ s^rcti'.! script is cct reqiiired. I'o try U cjAjjl^ t!:e entire 
crrL.t 5f -...T,d fil: .trii.. arJ to prciyce scr.e detaij {toth a:;dio and vinz;ai) 
*'T t»'€' fir',t se. cr^l franes. ^ 

I. ^jtliffe -Cif sc-lrt is ccnsistent with infcrraticn a-d srecificaticns 

•.rr.vi^ed. 

i] parents playlr.q both roles (i.e., roler ff child atjd 
~' rargnt 

L) ir;clu€ss ideas about visijal as iiell as audio ccntent 
c} eoldins fiirpcse/rea'^cn for role-playing 



d) tries to rut parents at ease/relates role-playing _ to 
tte-/ indicates that parents Tiay have trouble at first 
playing role of child fcut no need for er>arassr.ent 

e) sc-e explanaticn of toy: cuiTCses, ccntent, or use 
If. CjtliT.e of script demonstrates awareness of visual aspects of 

a) real use of visuals (i.e., visuals should convey i^ortant 
^" infonnation even without the audio; a ^reat deal of in- 
struction and irpact should be lost if subject's visuals 
i-;er2 caitted) 

b) awareness that this is fi Ins trip rather than rovie 

~ (i.e.. series of static shots rather than notion-oriented) 

III. The directions/instructions v;ere fonc>;2d. 

a) length: filn outlined should not be over five minutes 

or 50 frares 

b) entire ccntent outlined; enough detail (e.g., rough 
outline of ;^Jhat will be in narration and where it will 
cce) 

c) first several frares in considerable detail 
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3, Suggested Criteria and ffethods for Scoring. 

I. O-zerali rating of "adequate" or "inadequate." 

In order to receive an "adequate" ratinq, the subject inust receive 
checks on all the lines in I, 11, and III. 
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II. Z^%T^ of ^:degj3cy sccre— sccre ere point fcr eacfi of t^e oieclcs 
cojTited tOi?ard tfce 10 rat?aired fcr an **adEi3uate** rating- (C€tr»ert 
toint score inti5 equivalent cn 0-10 srale-) 

iil. Icv^ (of 3i: plied rele^^ant knowlefse) sccre 

3} rate t^.e subject's response as to the sccpe of reie^j^Tit 
Jcr.DwIedge brci^ght to bear cn the question cn a scale of 0-4 
(fron very liraited to very broad). You night want to use the 
Su1de1ir-es suggested under question #1. 

fc) exarples of possible "imaginative" responses 

Indication to oarent ttat role-played child should not be 
"icear cf.iiil/but should be "natural"* and allow prcblenis 
to occur; sor:e indication teat models used in fila will be 
siriilar to target audience; budget considerations; detailed 
instructions for visuals (e.g., close-ups, pan shots). 

£a-:pTe Responses and Suggested Scoring 

Tne following protocol, reproduced in its entirety (with 

coments about scoring keyed to nunbered parts of the response), should 

provide sore guidance as to possible criteria and scoring of task #4- 
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rar- 



i 
2 



6. 

7, 

3. 
9. 

19. 

n. 

12. 



¥£j have seen a parent and child playing 
ga!::es with ti'e feely Bag toy- 

Td ^lelp yoii learn to play gair^es with your 
child, yen are goicg to practice mth the 
toy here. 

Tnis practice is called role-playing. Gne 
of ycu is going to pretend to be a parent 
and anotJier is going to pretend to fee a 
child. 

Acting the role or part of a parent shouldn't 
be very hard, but acting the part of a child 
ray be hard at first- 

Here is a ran pretending to be a three-year- 
old child *fhile his "rrothar" shows hin a 
new gar:e. 

(Parent) ""that shape is called a triangle. 
Can you find a shape in the bag inrhich is 
the sare as the triangle?"* 

(Parent) "see if you can find the sair^e 
shape without peeking into the bag.** 



(«Child«) n found it.** 

(Parent) *No, that is a square. Can you 
find this shape?" 

C^ChiW) ""here it is," (Parent) ''Yes, 
these two shapes are the same." 



("Child-) '•Let's find all the ones with 
corners." (Parent) "Alright, you find 
then without peeking." 

("Child") " Is that all of them?" 
(Parent) "See if there are any more?" 



VISUR 
Feely Bag Toy- 
Parent and child playing 
with toy. 



Parent and child fron frar^e 
2 but in another pose. 



Exactly fraine 3 but with 
an adult in place of the 
child. 

"Child" picking yp a triangle 
frcn the table. 



Parent putting each of the 
4 shapes in the bag. 



"Child" peeking into bag 
while holding triangle out- 
side bag. 

"Child" holding up a square. 

"Parent" pointing at the 
triangle on the table. 

"Child" holding up a triangle 
and parent pointing at the 
two triangles. 

"Child" holding up a square. 



"Child" has two triangles, 
two squares, and one rectangle 
in front of him; he is putting 
second ^-ectangle on top of the 
other. 



3.J7 
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AVOID 



vism 



14- 
15. 



16 



17. 



13. 



19. 



^."C^iild") 'T^H dc-esn't haye corr.ers." 

(Parent) "Do you want to play another gare?" 
{-Child-) "TJo, r-n tired." 
(parent) "Alright, we'll step now. 

!<ow yen have seen how one parent pretended 
to be a child. He was role-playing. Did 
ym notice how the "child's" parent let fhe 
child change the rules of the gsse? That is 
ir-pcrtant. Also, the "child's" parent 
let the child step when he was tired. 

•Jow let's see what happens when these 
two people trade roles. 



("Cnild") "Tnese don't natch. 



(Parent) "That's right. Can you find one 
that ratches this one?" 

I.'otice that the parent is letting the 
child explore before starting the gaise. 



^-27 [The ga.-e is played twice* once to match 

{approx.) circles and once to natch triangles.] 

28. (Parent) "Can you find this shape, 

(approx.) a rectangle, in the bag?" 



29. 



30. 



("Child") "I don't want to." 
(Parent) "Alright, we won't play this 
game anymore today." 

Notice that the parent lets the child 
stop when she was tired. 

Mow you have seen two people take 
turns role-playing a parent and a child. 
Vlhen they do this, they find out what it 
feels like to be in "someone else's shoes." 
To help you learn how to use the Feely 
Bag toy with your child, you are ncrrs 
going to get a chance to act or role play 
the part of a child and the part of a 
parent. 



•"Child" holding up a circle. 

"•Cnild" locks bored and 
is locking under the table- 



Return to frar^e 11. 



Persons playing the parent 
and the child have inter- 
changed positions. 

"'Child" is putting a square 
on a rectangle. 

Parent points at the square- 



Child is fitting the two 
triangles together to make 
a parallelogram. 



[Suitable pict&res] 



Parent holding up rectangle 
from table. 

"Child" has turned away 
from the toy. 



Return to frame 16. 
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KEYED SCC-HiT^G GUIDE TO PROTOCOL F€R TASK 4 



*i]rier en Frarre nirt^r 

resccnse cliecklist In crotccol 



ia: 3 and 15 

Indicate that each p3*-ent has a turn playing 
■the child and the parent, so la receives a check. 

ib: The subject presents descriptions of the visual 

as well as the audio component for each frace, 
so receives a check for Ib- 



2, 3, and 30 

explain that role-playing the part of a child 
playing with Hie toy should help then under- 
stand problems and feelings that a child nay have 
in such a situation- This understanding should 
enable then to better help their child learn how 
to use the toy- (Ic receives a check-) 



Id: 4 

which demonstrates an awareness IJiat playing the 
role of a child may be difficult at first is an 
ifBDortant effort to relieve scsne of the 
enbarrassnient or tension parents might have about 
olaying the role of a child- (Id receives a check-) 

le- As specified in the instructions for the task, 

the film shows role-playing in the context of a 
parent and child playing with the Feely Bag toy- 
(I^ receives a check-) 

Ilg. Ila receives a check because visuals play a very 

important part in this film; sometimes the visual 
is an illustration of the accanpanying audio, 
sometimes the visual presents information that is 
not presented in the audio (e-g- in frame #9 the 
parent asks "Can you find this, shape?" While the 
visual shows her pointing to a triangle ), and 
SOTietimes the audio is a narration of what's 
happening in the accOTipanying visual (e-g., in 
frame 519 ^he child is sho-rfn fitting two triangles 
together while the audio has a narrator saying 
"Hotice tfiat the parent is letting the child explore 
before starting the game")- 

an another protocol, the visuals v/ere uninformative 
and contributed nothing to the film (e-g- "viev/ of 
class with one person speaking," "vierf of librarian 
standing"), so Ila would not receive a check- ) 

3.J9 
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response checklist In protocol 



The visuals indicate an understanding of the 
static frane nature of filnstrips l[e.g-, 
"child pointing," "child holding up square"), 
so lib receives a check. 

The length restriction has not been exceeded, 
so Ilia receives a check. 

The entire content of the film is outlined; 

the first igfranes in considerable detail 

and the last several in rough form (i.e- 

"the game is played t>;ice once to match circles 

and once to match triangles — with suitable 

pictures"), so I lib and IIIc each receive a check. 



This protocol received all 10 of the checks required for an "adequate" 
rating. In addition, several aspects of the response (e.g- frames 14 and 
29 in which the child gets bored, frames 15 and 19 which indicate that the child 
should be allowed to establish his/her own rules) show imagination and understanding 
of what would make the role-playing film most instructive- This response would 
be judged '^adequate" {a degree of adequacy score of 10) and might receive a scope 
score of "3." 
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Appendix F 
Miscellaneous Forms 





FOR FAl 
USE OTiLY, 

(iUL 

03 
05 

03 




EDJE Wmmti PFOGRAM BIOGRAPHIC DMA j 


Sex 


(circle cr.tr ^to:e) (/ears) 




EDICATICTJ: Hiqhest Degree {circle one): High School, 8A/5S, Kfl/MS, Ph.D., 

EdD. ] 


10 




The following is a list of courses that are frequently rienticned as 
useful in educational develcpi::ent, disse^aination and evaluation. Please 
-^ark by 3 nunber (0,1,2, etc.) the nunber of undergraduate, graduate or 
for^^al continuing education courses you have taken for each of the follow- 
ing. (Please place a zero (0) beside the category if you have taken no 
ccurses in this category.) 






Curriculun Developrent Journalism 


11 




Educational Psycholoqy Creative Kriting 


13 




Other Psychology Hedia Production (T.V. , film, etc.) 


15 


- 


Other Social Sciences Graphic Arts 


17 




Teacher Education Research Design 


19 




Conrnunication Theory Test & teasures, Psychone tries 


21 




Instructional Technoloqy Evaluation 


23 




Va rke t i ng S ta ti s ti cs 


25 




Administration Philosophy or Humanities 


27 


II. 


PROFESSIOIIAL MTOtRSHlr: Are you a ntsriDer or any uaviuuai |xruic:>:>iuiiai 

organizations? (circle one) YES NO 


29 


III. 


PUBLICATIONS AJID PRODUCTS: Please indicate the approximate number of items 
for which you have been the author or co-author. (If you have never produced 
a specific type of item, please place a zero (0) beside that category. 






Articles in professional journals Tests/Assessment Instruments 


30 




Technical R&D Reports Programmed Lessons 


34 




Evaluation Reports (of products/ Slide-Tapes/ Audio Tapes 
programs) 


OO 




Chapters in Professional Books Instructional Hims/l.y. 


42 




OfhPr Publications (Please Specify) Other Products IHiease :>neciTy; 


46 




( ) ( . _ — 1 








IV. 


TEACHING EXPERIENCE: Huve you ever taught? YES _ NO (circle one) 


50 




If YES, what kind of teaching? Student Substitute Regular 

(circle which one(s)) 

Hw many years? 


51 
52 



2 



^ i^-H%T >.C^' f ■VLniML: 'r^ .oj s-i-rreJitl*^ er^oyed? tlS Tio {circle or.*-) 



Erplcyer 



Fdjor Activities 



r.^-u jrCu ei^er dene any work (other than that described in Section V iir^ediately 
^Il'J^:) ifi .':hich jou ^ere involved in research, develcpcent, evaluation or disseai- 
r^tion-Jif fusion? YES VS (circle one) 

I'' t£S, Please complete the following, beginning with the nost RECEfJT position. 



1. Position Title^ 
Erployer^ 



Fron 



To 



^f^^ajor Activities^ 



2. Position Title^ 
Er:ployer 



From 



To 



Major Activities 



3.* Position Title_ 
Employer 



From 



To 



Major Activities^ 
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Please checX whiisi of the following DDSE InBtructi<»ial Moaoles you hanre 
coepleted. 

PLAJWRJG SERIES 
1.0 An Overview 



1.1 Problea Definition and Specification of Outccoes 

1.2 Consideration of AlteiT^tives 
PLANN1^^G/^EVELQF^EN^ SERIES 
1.5/4.1 Introduction to Develqsaent 
PLAJ^NHiG/EVAIjUATION SERIES 
1.4/5.1 Introduction to Evaluation 
PLA.V5I»iG/DiSSE!€NATia^ AND MMLKETDiG SERIES 
1.5/6.1 Introduction to Dissesaination ar.d Marketing 

INFOJKVriON/mTA a3LLECriON AND ORGANIZATION SERIES 

2.1 Orientation to Collecting and Organizing ED5E Infonnation and Data 

S.?!tetrilSl"of Infonnation Using Bibliogr^hic Resources 
2!4 The Retrieval of Infonnation Using Special Sources 

(3»MJNICATI0N SKILLS SERIES 

3.1 Oral CbCTHunication 

' 5.2 r Written Instructional Materials 

■ 5.3 Tedinical Writing: Infomal Docujnents 

' 3!4 Technical Writing: Fbnial Documents 

DB'ELOPMENT SERIES 

4.1/1.3 See Planning Series 

4.2 Designing Educational Products 

4.3 Product Engineering 

! 4.4 Review, Tryout, and Revision 
4.5 Special Problems in Developnent 



EVALUATION SERIES 

S. 1/1-4 See Planning Series 

5.2 MeasureiT^nt and Testing for Developers and Evaluators 

5.3 Desing of Evalijation Instrunients 

5.4 Planning and Iji?)leinenting Evaluation 

5.5 Evaluation Problems 

DISSBUNATION AND MARKETING SERIES 

6-1/1-5 See Planning Series 
6-2 Marketing Educational Products 
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